Review tracker
Where every item from the two outside reviews stands today: the 78 items in the QA review of 5 November 2026 and the 58 findings in the product review handed over on 9 November 2026. For what each feature does, see the feature list.
Waiting for Kombee Needs an account, key, price, decision or legal text from Kombee Partly done Live; one part waits for Kombee Please re-test Fixed; the reviewer should check it again Fixed Fixed and live on the test site Decided Works as designed; the decision is recorded No change needed Not a bug, or the reviewers’ own files
Checked against the code and the live site on 20 November 2026.
Still open, in one list
9 items across both reviews are not fully closed. Each one says what it is waiting for.
- QLB-01 Library is far too small for a HackerRank competitor: Partly done
- SEC-01 Employee’s ’hide my levels’ setting is ignored: Please re-test
- SEC-02 Privacy Notice and Terms are placeholders: Waiting for Kombee
- SEC-07 Data residency is India only: Waiting for Kombee
- PLT-01 Video, voice, SMS/WhatsApp and AI providers set to ’Stand-in (mock)’: Partly done
- PLT-04 Plan limits not enforced: Partly done
- PLT-05 Prices not final and inconsistent across currencies: Waiting for Kombee
- F-045 WhatsApp reminders and Talent Passport not in the spec: Partly done
- F-055 24 requirements couldn't be tested: email inbox for testing, SCIM, load test, SOC 2/DPA papers: Partly done
No item matches.
1. QA review, 5 November 2026 (78 items)
| ID | What was raised | Status | Where it stands |
|---|---|---|---|
| SEC-02 | Privacy Notice and Terms are placeholders High | Waiting for Kombee | The privacy notice and Terms need Kombee’s lawyer. A draft data processing outline and sub-processor list are in KOMBEE-DECISIONS. |
| SEC-07 | Data residency is India only Low | Waiting for Kombee | Data is stored in Mumbai (India). A Gulf region is a Kombee decision with a running cost, once a Gulf client needs it. |
| PLT-05 | Prices not final and inconsistent across currencies Medium | Waiting for Kombee | The prices on screen are samples. Waiting for Kombee’s price list. |
| QLB-01 | Library is far too small for a HackerRank competitor High | Partly done | The library has 2,346 questions: 142 reviewed and 2,204 written by AI and marked “AI draft”. Since 19 Nov AI drafts can’t be used in tests until a Kombee reviewer approves them (Console → Library review). Waiting for Kombee to name reviewers. |
| PLT-01 | Video, voice, SMS/WhatsApp and AI providers set to ’Stand-in (mock)’ High | Partly done | AI, email, video (LiveKit), speech (Deepgram) and the avatar (Tavus, on its free plan) are real on the live site. WhatsApp waits for Kombee’s Meta business portfolio; SMS and calendars stay stand-ins until those accounts exist. The status and health pages say so. |
| PLT-04 | Plan limits not enforced Medium | Partly done | Plan limits are built and live, but switched off for companies with no plan while testing, so the test companies aren’t capped at 10 tests a month. Switch on after giving them a plan. |
| SEC-01 | Employee’s ’hide my levels’ setting is ignored High | Please re-test | Fixed in the code and tested. The live setting was “show” when we checked, so the hide path needs a re-test on the live site. |
| INT-01 | One attempt shows different integrity scores on different screens Critical | Fixed | One integrity score from one calculation, the same on the profile, compare, CSV export and printable report. |
| INT-02 | A name mismatch at the start is recorded but not surfaced High | Fixed | A name that doesn’t match the invitation is now flagged on the candidate page and the integrity case. |
| INT-03 | Full-screen and monitoring gate is enforced only in the browser High | Fixed | The server checks that monitoring was set up before it accepts answers, not only the browser. |
| INT-04 | Timer starts before monitoring is set up, and the set-up repeats per section High | Fixed | Set-up time is given back, and monitoring carries over between sections instead of starting again. |
| INT-05 | No fallback on the start screen when full screen is refused Medium | Fixed | If full screen is refused, the start screen explains what to do and offers to carry on (recorded). |
| INT-06 | Candidate can be advanced while an integrity case is open High | Fixed | A candidate with an open integrity case can’t be advanced until a person decides the case. |
| INT-07 | Code replay is snapshot-only; pasting a whole solution isn’t flagged Medium | Fixed | Typing replay records changes as they happen; pasting a large block is flagged with its size. |
| INT-08 | AI integrity summary contradicts the page Low | Fixed | The AI case summary is written from the same evidence the page shows, and is labelled as AI. |
| INT-09 | Candidate gets no warning when they leave the test tab Low | Fixed | Candidates see a one-line notice the first time they leave the test tab. |
| DEC-01 | ’Advance together’ advances candidates with no confirmation, no reason and no evidence Critical | Fixed | “Advance together” asks for confirmation and a reason, and only includes candidates with results. |
| DEC-02 | Decision radio is pre-selected and the reason is optional Critical | Fixed | Nothing is pre-selected and every decision needs a reason. |
| DEC-03 | Integrity decision pre-filled: ’Clear’ selected and the candidate-facing reason already typed High | Fixed | Integrity decisions start empty: the reviewer chooses and writes the reason. |
| DEC-04 | AI hiring suggestion is slow, silent, and goes stale Medium | Fixed | The AI suggestion shows that it is working, and says when newer results make it out of date. |
| DEC-05 | AI interview summary duplicated on the profile Low | Fixed | The AI interview summary appears once on the profile. |
| AIQ-01 | AI safety check returns ’no clear answer’ and breaks three AI features Critical | Fixed | The AI safety check asks for a short JSON answer and asks once more if it can’t read it. Checked on the live server with the real model. |
| AIQ-02 | ’Ask the AI how hard it is’ shows confidence 0% Low | Fixed | “Ask the AI how hard it is” shows a real confidence figure. |
| CAN-01 | SQL questions can’t be run before submitting Medium | Fixed | SQL answers can be run before submitting. |
| CAN-02 | MCQ results don’t show which option the candidate chose Medium | Fixed | Multiple-choice results show which option the candidate chose. |
| CAN-03 | ’Marked for review’ list is empty on submit Low | Fixed | The “Marked for review” list fills in on submit. |
| CAN-04 | Practice page contradicts itself Low | Fixed | The practice page wording now agrees with itself. |
| CAN-05 | Instructions contradict each other Low | Fixed | Test instructions were rewritten so they no longer contradict each other. |
| CAN-06 | Markdown not rendered in question text Low | Fixed | Question text shows formatting (Markdown) properly. |
| CAN-07 | Written question shows rubric headings to the candidate Low | Fixed | Rubric headings are hidden from candidates on written questions. |
| CAN-08 | Timers and counts are confusing Low | Fixed | Timers and counts were relabelled and made consistent. |
| CAN-09 | Run results below the fold; mouse wheel over editors doesn’t scroll the page Low | Fixed | Run results appear in view, and the mouse wheel scrolls the page over editors. |
| CAN-10 | Candidate progress tracker marks ’Interview’ done too early Low | Fixed | The progress tracker marks “Interview” done only when the interview is over. |
| CAN-11 | Template ’AI-assisted developer’ shows ’AI not allowed’ badge Low | Fixed | The “AI-assisted developer” template shows the right AI badge. |
| GRD-01 | Rubric points don’t add up to the question’s points Medium | Fixed | Rubric points must add up to the question’s points before it can be saved. |
| GRD-02 | Provisional score shown like a final score Medium | Fixed | A score still waiting for marking is shown as provisional. |
| GRD-03 | Skill levels overclaim from tiny evidence Medium | Fixed | Skill levels take the amount of evidence into account and say when it is thin. |
| GRD-04 | Printable candidate report is too thin to share Medium | Fixed | The printable report now has the full breakdown, integrity summary and decisions. |
| GRD-05 | Time and count statistics round to zero or disagree Low | Fixed | Time and count figures no longer round to zero or disagree. |
| BLD-01 | A failed save wipes the unsaved edits in the same panel Medium | Fixed | A failed save keeps the edits in the panel. Test added. |
| BLD-02 | JD draft gives every question 10 points and mismatched reasons Medium | Fixed | A test drafted from a job description gives each question sensible points and its own reason. |
| BLD-03 | JD draft defaults are unsafe; publish has no confirmation Medium | Fixed | Safer defaults on job-description drafts, and publishing asks for confirmation. |
| BLD-04 | Invalid points silently ignored; totals go stale Low | Fixed | Invalid points are refused with a message. We could not reproduce the stale total from the code: please re-check. |
| BLD-05 | Declaration stores raw HTML Low | Fixed | Declarations are stored as plain text. |
| BLD-06 | Invite is discoverable only by scrolling a long popover Low | Fixed | Invite options are in tabs, and the first click works. |
| QLB-02 | Variants made on a SetBenchmark library question from a company account Medium | Fixed | Working as designed: variants of a library question belong to the company that made them. |
| QLB-03 | ’Version A / Version B’ in the builder vs ’No variants yet’ in Shield Low | Fixed | The builder and Question Shield now use the same words for variants. |
| QLB-04 | Wrong message when sending a question without a reference solution Low | Fixed | The message now says the question needs a reference solution. |
| QLB-05 | Question marking table hides test weights and hidden flags Low | Fixed | The marking table shows each test’s weight and whether it is hidden. |
| IVW-01 | Interviewer is locked out of their own interview and scorecard High | Fixed | Interviewers can open their own interviews and scorecards. |
| IVW-02 | Live video is simulated; the old seeded room errors High | Fixed | Live video is real: LiveKit Cloud is switched on and its check passes, and recordings go to the files bucket. The old sample room was removed. Still to try: a first real interview with recording. |
| IVW-03 | Interviews never end on their own Medium | Fixed | Interviews end on their own after the planned time. |
| IVW-04 | ’Calendar invites have been sent’ when nothing was sent Medium | Fixed | The message only says invites were sent when a calendar is connected and they really were. |
| IVW-05 | Small interview issues Low | Fixed | The small interview issues listed were fixed. |
| PIP-01 | Duplicate candidates with the same email are accepted silently High | Fixed | Adding a candidate with an email that is already there shows the existing candidate instead. |
| PIP-02 | Candidates page crashes with an unexpected filter value Medium | Fixed | Unknown filter values are ignored instead of crashing the page. |
| PIP-03 | Problem reports from candidates aren’t queued anywhere Medium | Fixed | Problems that candidates report go to a queue with “Mark handled”. |
| PIP-04 | Adjust form can’t grant what candidates ask for Medium | Fixed | The adjustment form can grant everything candidates can ask for. |
| PIP-05 | Privacy request didn’t record on first submit Low | Fixed | Forms wait for the page to finish loading before they can be sent. |
| PIP-06 | ’Copy link for a colleague’ gives no feedback Low | Fixed | “Copy link for a colleague” confirms that the link was copied. |
| ROL-01 | Interviewer and Hiring manager have more access than they need Medium | Fixed | Interviewer and hiring manager roles were cut down to what they need. |
| ROL-02 | Super admin needs a separate sign-in that expires within the hour Low | Fixed | “View as” explains that Super admin needs its own sign-in. |
| SEC-03 | Integrity status sent to Greenhouse and Lever by default High | Fixed | Integrity results are sent to Greenhouse and Lever only if the company switches it on. |
| SEC-04 | Missing browser security headers Medium | Fixed | All pages send a full content security policy and the other security headers (live since 5 Nov). |
| SEC-05 | Full API reference is public, including internal and dev-only endpoints Medium | Fixed | Internal and development routes are gone from the public API reference, and every route has a description. |
| SEC-06 | ’Sandbox’ API keys act on real data Medium | Fixed | A sandbox API key works in its own hidden company, so it can’t reach real data. |
| PLT-02 | No working self-serve company sign-up High | Fixed | Self sign-up is live: “Start free trial” on setbenchmark.com and its pricing page opens sign-up with a work email, and the new workspace opens on “Get started”. The final Terms (SEC-02) and branded sign-in on Kombee’s own domain (PLT-03) are tracked on their own. |
| PLT-03 | Sign-in and sign-up pages are an unbranded staging page Medium | Fixed | Sign-in and sign-up run on the production WorkOS account, with SetBenchmark’s name and logo and no staging warnings; email and Google sign-in are tested. Kombee’s own sign-in domain is not for now (F-024) and can be switched on later without code. |
| PLT-06 | Status page says ’Everything is working’ while services are mocked Medium | Fixed | The status page checks the code runner and email live, and says “Not connected” for parts not set up. |
| PLT-07 | Small console issues Low | Fixed | The small console issues listed were fixed. |
| ANA-01 | ’Completion’ column shows the average score Medium | Fixed | The “Completion” column shows completion. |
| ANA-02 | Funnel and time metrics look wrong Medium | Fixed | Funnel and time figures were corrected. |
| ANA-03 | Team skill map summary contradicts its data Low | Fixed | The team skill map summary is worked out from the same data as the map. |
| UIX-01 | Many controls have no useful accessible name Medium | Fixed | Controls have proper accessible names. A WCAG 2.2 AA scan now runs on every build. |
| UIX-02 | Lower-case names and inconsistent labels Low | Fixed | Names and labels were made consistent. |
| UIX-03 | Search returns unrelated results Low | Fixed | Search ranks closer matches first and suggests results as you type. |
| UIX-04 | Audit log uses internal names Low | Fixed | The audit log uses plain names. |
| UIX-05 | Truncation at laptop and phone widths Low | Fixed | Long names wrap or shorten neatly at laptop and phone widths. |
2. Product review handoff, 9 November 2026 (58 findings)
“Decided” items were open questions for Kombee; they were decided on 12 November with reasons (KOMBEE-DECISIONS), and Kombee can change any of them.
| ID | What was raised | Status | Where it stands |
|---|---|---|---|
| F-045 | WhatsApp reminders and Talent Passport not in the spec | Partly done | The Talent Passport is built. WhatsApp reminders are built and wait for Kombee’s Meta business account. |
| F-055 | 24 requirements couldn't be tested: email inbox for testing, SCIM, load test, SOC 2/DPA papers | Partly done | Load test done on 13 Nov: about 150 candidates at once on today’s server; 500 need a 4-CPU server (Kombee to approve the cost). SOC 2 is outside the agreed scope; a draft DPA outline is in KOMBEE-DECISIONS. Testing email goes to the stand-in outbox on a test copy. |
| F-001 | Switching to "Candidate" in Chrome fails | Fixed | Security setting changed to allow our own candidate site and console. Checked on the live site. |
| F-002 | When testing a question, a wrong answer only says "failed t1" with no error details | Fixed | Authors (when testing a question) and recruiters (on the candidate page) now see, for each failing test, the input, expected and actual output and the error, hidden tests included. Candidates still see only their examples. |
| F-004 | Python editor starts empty; code that defines a function but doesn't read input prints nothing | Fixed | When a question has no starter code for a language, the editor starts with code that reads the input, in 12 languages. Authors can still write their own. |
| F-005 | A retired question can't be brought back | Fixed | Reviewers have "Bring back": the question is live again with its approved version (or a draft if never approved). |
| F-008 | Security warnings on the WorkOS sign-in page | Fixed | Sign-in now runs on the production WorkOS account, so its test-page warnings are gone. Email and Google sign-in are live and tested; Microsoft is hidden until its app is set up. The sign-in page stays on WorkOS's own address for now (no custom domain). |
| F-009 | A link on the Appearance page relies on colour only | Fixed | Links inside text are now underlined everywhere. |
| F-010 | A scrolling strip on the Appearance page can't be reached by keyboard | Fixed | The sample table in the preview no longer scrolls. |
| F-011 | Links on the candidate join and instructions pages rely on colour only | Fixed | Same fix as F-009. |
| F-012 | 18 smaller accessibility issues (missing header area, heading order) | Fixed | Top bar marked as the page header, the candidate join page has a page title, and heading levels on the marking page are in order. A re-scan is in the regression checklist (B43). |
| F-017 | No Coding Practice area | Fixed | Live at /practice (see scope item 1). |
| F-019 | No competitions, hackathons or leaderboards | Fixed | Companies run competitions and hackathons on their own tests (Competitions in the menu): individuals or teams, an AI-drafted option that a person must approve, live standings, a copying check where a person clears or disqualifies each flagged entry, and a signed rank certificate with a badge and prize for every finisher. Public developer contests in the community get live standings and rank certificates too. |
| F-020 | AI Fluency, Certifications, SCIM user sync and the "agentic IDE" not seen | Fixed | AI Fluency: a 0–100 rating in four parts (prompting, critique, verification, result) on every test that allows AI, replaced by the reviewer's marks once given and added to the skill profile and Passport. Certifications: exams on their own question pool, digitally signed certificates, renewal with a reminder. SCIM: directory sync for Okta, Entra ID and Google (Settings → Sign-in). Agentic IDE: a terminal and a live preview beside the coding agent, with every command recorded for the reviewer. |
| F-022 | Shareable test link doesn't show when it expires | Fixed | Shows the closing date with the time left ("3 days left"), or "no closing date: it works until you turn it off". |
| F-025 | Question authors get a plain text box, not a code editor | Fixed | The candidates' code editor is used for answers, starter code and reference solutions. |
| F-026 | The code box is cramped on phones | Fixed | Comes with F-025 (the editor sizes itself to the screen). |
| F-027 | Authors can't run a question against their own input | Fixed | "Test against my own input" added under the answer. |
| F-028 | A candidate's invitation link works only once; reopening it locks them out | Fixed | Links stay single-use (security), and a candidate whose link no longer works now types their email under "Email me a new link": a new link goes only to the address the company has, only while a test is open, with the same answer either way. |
| F-031 | "Notice period" and typing your full name are required before starting, with no explanation | Fixed | A line under the button already says what is missing ("Type your name to start"). Which details are required is set by each company per test. |
| F-032 | The main address goes to sign-in; no public sign-up | Fixed | The main address now has public pages (home, pricing) with “Start free trial”, and self-service sign-up is live. Sign-in on Kombee’s own domain waits for the production WorkOS account (F-024). |
| F-033 | Should "View as Candidate" start a real interactive test? | Fixed | It opens a fresh sample candidate's real test (works now after the F-001 fix). |
| F-047 | Turning off a sign-in method may not sign out people already signed in | Fixed | Anyone signed in that way is now signed out at once; the admin making the change stays in. |
| F-050 | AI Fluency thresholds not defined | Fixed | 75 and up Strong, 50–74 Developing, under 50 Needs support (KOMBEE-DECISIONS section 6). |
| F-051 | "Save for everyone" may stay disabled when only switching light/dark | Fixed | Those buttons only choose which colours to edit (nothing to save yet). Renamed "Edit light colours" / "Edit dark colours" with a note saying so. |
| F-052 | Clearing the brand name falls back to the company code without explaining | Fixed | The hint now says what is used when it is left empty. |
| F-053 | API keys page doesn't show the API address or a docs link | Fixed | It now shows both. |
| F-006 | Custom roles don't appear in "View as" | Decided | Decided 12 Nov: No. "View as" is a demo tool and will be removed before real use (you asked to keep it until testing ends). |
| F-007 | "View as" for staff roles changes the view only | Decided | Decided 12 Nov: Acceptable. Each staff role in "View as" signs in as a real sample user of that role, with that role's permissions. Only "Candidate" opens a separate site. |
| F-018 | Only 3 programming languages offered, not 20 | Decided | Decided: 12 languages at launch (Python, JavaScript, TypeScript, Java, C, C++, C#, Go, Rust, Kotlin, Ruby, PHP), more on request. New coding questions start with 7 ticked, each with starter code. |
| F-021 | API keys say "sandbox/live", the spec says "test/production" | Decided | Decided 12 Nov: Keep "sandbox/live". |
| F-023 | No one-click download on the analytics number cards | Decided | Decided 12 Nov: No; table downloads are enough. Full downloads (spreadsheet and PDF) exist on the analytics tables. |
| F-024 | Sign-in goes to a WorkOS address without warning | Decided | Decided 20 Nov: not for now. The production WorkOS sign-in page carries SetBenchmark's name and logo but stays on WorkOS's address; a custom sign-in domain costs about $99 a month. Revisit if buyers object. |
| F-030 | Tests only run in Chrome or Edge, so Firefox and Safari users are blocked | Decided | Decided 12 Nov: Keep Chrome/Edge only, and say so up front. Done on purpose: monitoring and voice need features only Chrome and Edge have. |
| F-034 | "Try the editor first" practice before the timed test | Decided | Decided 12 Nov: Keep. It reduces nerves and avoidable complaints. |
| F-035 | Rejecting and blocking a candidate needs two people | Decided | Decided 12 Nov: On by default; a company may switch it off. Done on purpose (a fairness safeguard). |
| F-036 | Two timers (section and whole test) shown together | Decided | Decided 12 Nov: Keep both. The section timer helps candidates pace each part; the overall timer shows the hard end. |
| F-037 | A− / A+ text size buttons for candidates | Decided | Decided 12 Nov: Keep. |
| F-038 | Names hidden while marking (blind marking) by default | Decided | Decided 12 Nov: Keep. It reduces bias. |
| F-039 | Module names "Employee skills" and "Bench" vs "SkillUp" in the spec | Decided | Decided 12 Nov: Keep the screen names. |
| F-040 | AI policy labels differ from the spec | Decided | Decided 12 Nov: Keep "AI not allowed / allowed / required". |
| F-041 | "Find developers" marked Pilot, with an email introduction step | Decided | Decided 12 Nov: Yes, pilot at launch. |
| F-042 | Vendor portal shows agency proxy rates and a watch list | Decided | Decided 12 Nov: Keep, worded as indicators to look into, not accusations; visible only to the company's own staff. This is scope item 2. Kombee's legal team should approve the wording shown about agencies. |
| F-043 | Bench sends reassessment invitations to employees automatically | Decided | Decided 12 Nov: HR sends them deliberately; no second approver. Small change. |
| F-044 | Question Shield is not in the spec | Decided | Decided 12 Nov: Yes, add it. It protects the question bank. |
| F-048 | Which skill rating wins on the bench (manager, test or self-rating)? | Decided | Decided 12 Nov: Tested score only; claimed skills show as "Stated" and count for nothing until tested. All three are shown. |
| F-049 | How question versions work isn't defined | Decided | Decided 12 Nov: Published versions are fixed; edits make a new draft; tests keep the version they were built with. Each published version is fixed and tests use the version they were built with. Edits make a new draft. |
| F-054 | About 50 QA test candidates can't be deleted | Decided | Decided: yes. 96 QA test records were erased on 12 Nov through the privacy service, with deletion certificates. |
| F-056 | 16 places where the BRD and FRD disagree | Decided | Decided 12 Nov: Kombee issues v1.1 of the BRD and FRD; the corrections are drafted in KOMBEE-DECISIONS section 2. |
| F-003 | "Finish section" box may block clicks on the header | No change needed | Checked: it is a proper modal box, which blocks the page behind it on purpose until "Keep working", Esc, or "Finish section". |
| F-013 | Switching to Candidate signs the admin out | No change needed | The reviewers could not reproduce it. |
| F-014 | API addresses return "not found" on the website | No change needed | Withdrawn by the reviewers: the API has its own address. |
| F-015 | Duplicate email warning when adding a candidate | No change needed | Withdrawn by the reviewers. |
| F-016 | Old bug labels in a draft report | No change needed | The reviewers' own paperwork. Nothing to change. |
| F-029 | "Add a candidate" is hidden inside a folded section | No change needed | Checked: it is already a visible primary button that opens a panel; screen readers announce it as one. |
| F-046 | The on-screen timer pauses while the tab is hidden, then corrects itself | No change needed | The server keeps the real time. The screen catches up when you return, and no time is lost or gained. |
| F-057 | The reviewers' own automated tests: 17 always pass and 17 check the wrong thing | No change needed | These are in the reviewers' test files, not in SetBenchmark. Their team should fix them. Our own test suite (2,114 tests) passes. |
| F-058 | A reviewer report quoted performance numbers that aren't in the spec | No change needed | The reviewers’ own paperwork. Numeric targets have since been agreed (KOMBEE-DECISIONS section 3). |