Review tracker

Where every item from the two outside reviews stands today: the 78 items in the QA review of 5 November 2026 and the 58 findings in the product review handed over on 9 November 2026. For what each feature does, see the feature list.

Waiting for Kombee Needs an account, key, price, decision or legal text from Kombee Partly done Live; one part waits for Kombee Please re-test Fixed; the reviewer should check it again Fixed Fixed and live on the test site Decided Works as designed; the decision is recorded No change needed Not a bug, or the reviewers’ own files

Checked against the code and the live site on 20 November 2026.

Still open, in one list

9 items across both reviews are not fully closed. Each one says what it is waiting for.

1. QA review, 5 November 2026 (78 items)

71Closed (fixed, decided or no change needed)
4Partly done or to re-test
3Waiting for Kombee
78Items in all
IDWhat was raisedStatusWhere it stands
SEC-02Privacy Notice and Terms are placeholders
High
Waiting for KombeeThe privacy notice and Terms need Kombee’s lawyer. A draft data processing outline and sub-processor list are in KOMBEE-DECISIONS.
SEC-07Data residency is India only
Low
Waiting for KombeeData is stored in Mumbai (India). A Gulf region is a Kombee decision with a running cost, once a Gulf client needs it.
PLT-05Prices not final and inconsistent across currencies
Medium
Waiting for KombeeThe prices on screen are samples. Waiting for Kombee’s price list.
QLB-01Library is far too small for a HackerRank competitor
High
Partly doneThe library has 2,346 questions: 142 reviewed and 2,204 written by AI and marked “AI draft”. Since 19 Nov AI drafts can’t be used in tests until a Kombee reviewer approves them (Console → Library review). Waiting for Kombee to name reviewers.
PLT-01Video, voice, SMS/WhatsApp and AI providers set to ’Stand-in (mock)’
High
Partly doneAI, email, video (LiveKit), speech (Deepgram) and the avatar (Tavus, on its free plan) are real on the live site. WhatsApp waits for Kombee’s Meta business portfolio; SMS and calendars stay stand-ins until those accounts exist. The status and health pages say so.
PLT-04Plan limits not enforced
Medium
Partly donePlan limits are built and live, but switched off for companies with no plan while testing, so the test companies aren’t capped at 10 tests a month. Switch on after giving them a plan.
SEC-01Employee’s ’hide my levels’ setting is ignored
High
Please re-testFixed in the code and tested. The live setting was “show” when we checked, so the hide path needs a re-test on the live site.
INT-01One attempt shows different integrity scores on different screens
Critical
FixedOne integrity score from one calculation, the same on the profile, compare, CSV export and printable report.
INT-02A name mismatch at the start is recorded but not surfaced
High
FixedA name that doesn’t match the invitation is now flagged on the candidate page and the integrity case.
INT-03Full-screen and monitoring gate is enforced only in the browser
High
FixedThe server checks that monitoring was set up before it accepts answers, not only the browser.
INT-04Timer starts before monitoring is set up, and the set-up repeats per section
High
FixedSet-up time is given back, and monitoring carries over between sections instead of starting again.
INT-05No fallback on the start screen when full screen is refused
Medium
FixedIf full screen is refused, the start screen explains what to do and offers to carry on (recorded).
INT-06Candidate can be advanced while an integrity case is open
High
FixedA candidate with an open integrity case can’t be advanced until a person decides the case.
INT-07Code replay is snapshot-only; pasting a whole solution isn’t flagged
Medium
FixedTyping replay records changes as they happen; pasting a large block is flagged with its size.
INT-08AI integrity summary contradicts the page
Low
FixedThe AI case summary is written from the same evidence the page shows, and is labelled as AI.
INT-09Candidate gets no warning when they leave the test tab
Low
FixedCandidates see a one-line notice the first time they leave the test tab.
DEC-01’Advance together’ advances candidates with no confirmation, no reason and no evidence
Critical
Fixed“Advance together” asks for confirmation and a reason, and only includes candidates with results.
DEC-02Decision radio is pre-selected and the reason is optional
Critical
FixedNothing is pre-selected and every decision needs a reason.
DEC-03Integrity decision pre-filled: ’Clear’ selected and the candidate-facing reason already typed
High
FixedIntegrity decisions start empty: the reviewer chooses and writes the reason.
DEC-04AI hiring suggestion is slow, silent, and goes stale
Medium
FixedThe AI suggestion shows that it is working, and says when newer results make it out of date.
DEC-05AI interview summary duplicated on the profile
Low
FixedThe AI interview summary appears once on the profile.
AIQ-01AI safety check returns ’no clear answer’ and breaks three AI features
Critical
FixedThe AI safety check asks for a short JSON answer and asks once more if it can’t read it. Checked on the live server with the real model.
AIQ-02’Ask the AI how hard it is’ shows confidence 0%
Low
Fixed“Ask the AI how hard it is” shows a real confidence figure.
CAN-01SQL questions can’t be run before submitting
Medium
FixedSQL answers can be run before submitting.
CAN-02MCQ results don’t show which option the candidate chose
Medium
FixedMultiple-choice results show which option the candidate chose.
CAN-03’Marked for review’ list is empty on submit
Low
FixedThe “Marked for review” list fills in on submit.
CAN-04Practice page contradicts itself
Low
FixedThe practice page wording now agrees with itself.
CAN-05Instructions contradict each other
Low
FixedTest instructions were rewritten so they no longer contradict each other.
CAN-06Markdown not rendered in question text
Low
FixedQuestion text shows formatting (Markdown) properly.
CAN-07Written question shows rubric headings to the candidate
Low
FixedRubric headings are hidden from candidates on written questions.
CAN-08Timers and counts are confusing
Low
FixedTimers and counts were relabelled and made consistent.
CAN-09Run results below the fold; mouse wheel over editors doesn’t scroll the page
Low
FixedRun results appear in view, and the mouse wheel scrolls the page over editors.
CAN-10Candidate progress tracker marks ’Interview’ done too early
Low
FixedThe progress tracker marks “Interview” done only when the interview is over.
CAN-11Template ’AI-assisted developer’ shows ’AI not allowed’ badge
Low
FixedThe “AI-assisted developer” template shows the right AI badge.
GRD-01Rubric points don’t add up to the question’s points
Medium
FixedRubric points must add up to the question’s points before it can be saved.
GRD-02Provisional score shown like a final score
Medium
FixedA score still waiting for marking is shown as provisional.
GRD-03Skill levels overclaim from tiny evidence
Medium
FixedSkill levels take the amount of evidence into account and say when it is thin.
GRD-04Printable candidate report is too thin to share
Medium
FixedThe printable report now has the full breakdown, integrity summary and decisions.
GRD-05Time and count statistics round to zero or disagree
Low
FixedTime and count figures no longer round to zero or disagree.
BLD-01A failed save wipes the unsaved edits in the same panel
Medium
FixedA failed save keeps the edits in the panel. Test added.
BLD-02JD draft gives every question 10 points and mismatched reasons
Medium
FixedA test drafted from a job description gives each question sensible points and its own reason.
BLD-03JD draft defaults are unsafe; publish has no confirmation
Medium
FixedSafer defaults on job-description drafts, and publishing asks for confirmation.
BLD-04Invalid points silently ignored; totals go stale
Low
FixedInvalid points are refused with a message. We could not reproduce the stale total from the code: please re-check.
BLD-05Declaration stores raw HTML
Low
FixedDeclarations are stored as plain text.
BLD-06Invite is discoverable only by scrolling a long popover
Low
FixedInvite options are in tabs, and the first click works.
QLB-02Variants made on a SetBenchmark library question from a company account
Medium
FixedWorking as designed: variants of a library question belong to the company that made them.
QLB-03’Version A / Version B’ in the builder vs ’No variants yet’ in Shield
Low
FixedThe builder and Question Shield now use the same words for variants.
QLB-04Wrong message when sending a question without a reference solution
Low
FixedThe message now says the question needs a reference solution.
QLB-05Question marking table hides test weights and hidden flags
Low
FixedThe marking table shows each test’s weight and whether it is hidden.
IVW-01Interviewer is locked out of their own interview and scorecard
High
FixedInterviewers can open their own interviews and scorecards.
IVW-02Live video is simulated; the old seeded room errors
High
FixedLive video is real: LiveKit Cloud is switched on and its check passes, and recordings go to the files bucket. The old sample room was removed. Still to try: a first real interview with recording.
IVW-03Interviews never end on their own
Medium
FixedInterviews end on their own after the planned time.
IVW-04’Calendar invites have been sent’ when nothing was sent
Medium
FixedThe message only says invites were sent when a calendar is connected and they really were.
IVW-05Small interview issues
Low
FixedThe small interview issues listed were fixed.
PIP-01Duplicate candidates with the same email are accepted silently
High
FixedAdding a candidate with an email that is already there shows the existing candidate instead.
PIP-02Candidates page crashes with an unexpected filter value
Medium
FixedUnknown filter values are ignored instead of crashing the page.
PIP-03Problem reports from candidates aren’t queued anywhere
Medium
FixedProblems that candidates report go to a queue with “Mark handled”.
PIP-04Adjust form can’t grant what candidates ask for
Medium
FixedThe adjustment form can grant everything candidates can ask for.
PIP-05Privacy request didn’t record on first submit
Low
FixedForms wait for the page to finish loading before they can be sent.
PIP-06’Copy link for a colleague’ gives no feedback
Low
Fixed“Copy link for a colleague” confirms that the link was copied.
ROL-01Interviewer and Hiring manager have more access than they need
Medium
FixedInterviewer and hiring manager roles were cut down to what they need.
ROL-02Super admin needs a separate sign-in that expires within the hour
Low
Fixed“View as” explains that Super admin needs its own sign-in.
SEC-03Integrity status sent to Greenhouse and Lever by default
High
FixedIntegrity results are sent to Greenhouse and Lever only if the company switches it on.
SEC-04Missing browser security headers
Medium
FixedAll pages send a full content security policy and the other security headers (live since 5 Nov).
SEC-05Full API reference is public, including internal and dev-only endpoints
Medium
FixedInternal and development routes are gone from the public API reference, and every route has a description.
SEC-06’Sandbox’ API keys act on real data
Medium
FixedA sandbox API key works in its own hidden company, so it can’t reach real data.
PLT-02No working self-serve company sign-up
High
FixedSelf sign-up is live: “Start free trial” on setbenchmark.com and its pricing page opens sign-up with a work email, and the new workspace opens on “Get started”. The final Terms (SEC-02) and branded sign-in on Kombee’s own domain (PLT-03) are tracked on their own.
PLT-03Sign-in and sign-up pages are an unbranded staging page
Medium
FixedSign-in and sign-up run on the production WorkOS account, with SetBenchmark’s name and logo and no staging warnings; email and Google sign-in are tested. Kombee’s own sign-in domain is not for now (F-024) and can be switched on later without code.
PLT-06Status page says ’Everything is working’ while services are mocked
Medium
FixedThe status page checks the code runner and email live, and says “Not connected” for parts not set up.
PLT-07Small console issues
Low
FixedThe small console issues listed were fixed.
ANA-01’Completion’ column shows the average score
Medium
FixedThe “Completion” column shows completion.
ANA-02Funnel and time metrics look wrong
Medium
FixedFunnel and time figures were corrected.
ANA-03Team skill map summary contradicts its data
Low
FixedThe team skill map summary is worked out from the same data as the map.
UIX-01Many controls have no useful accessible name
Medium
FixedControls have proper accessible names. A WCAG 2.2 AA scan now runs on every build.
UIX-02Lower-case names and inconsistent labels
Low
FixedNames and labels were made consistent.
UIX-03Search returns unrelated results
Low
FixedSearch ranks closer matches first and suggests results as you type.
UIX-04Audit log uses internal names
Low
FixedThe audit log uses plain names.
UIX-05Truncation at laptop and phone widths
Low
FixedLong names wrap or shorten neatly at laptop and phone widths.

2. Product review handoff, 9 November 2026 (58 findings)

56Closed (fixed, decided or no change needed)
2Partly done or to re-test
0Waiting for Kombee
58Items in all

“Decided” items were open questions for Kombee; they were decided on 12 November with reasons (KOMBEE-DECISIONS), and Kombee can change any of them.

IDWhat was raisedStatusWhere it stands
F-045WhatsApp reminders and Talent Passport not in the specPartly doneThe Talent Passport is built. WhatsApp reminders are built and wait for Kombee’s Meta business account.
F-05524 requirements couldn't be tested: email inbox for testing, SCIM, load test, SOC 2/DPA papersPartly doneLoad test done on 13 Nov: about 150 candidates at once on today’s server; 500 need a 4-CPU server (Kombee to approve the cost). SOC 2 is outside the agreed scope; a draft DPA outline is in KOMBEE-DECISIONS. Testing email goes to the stand-in outbox on a test copy.
F-001Switching to "Candidate" in Chrome failsFixedSecurity setting changed to allow our own candidate site and console. Checked on the live site.
F-002When testing a question, a wrong answer only says "failed t1" with no error detailsFixedAuthors (when testing a question) and recruiters (on the candidate page) now see, for each failing test, the input, expected and actual output and the error, hidden tests included. Candidates still see only their examples.
F-004Python editor starts empty; code that defines a function but doesn't read input prints nothingFixedWhen a question has no starter code for a language, the editor starts with code that reads the input, in 12 languages. Authors can still write their own.
F-005A retired question can't be brought backFixedReviewers have "Bring back": the question is live again with its approved version (or a draft if never approved).
F-008Security warnings on the WorkOS sign-in pageFixedSign-in now runs on the production WorkOS account, so its test-page warnings are gone. Email and Google sign-in are live and tested; Microsoft is hidden until its app is set up. The sign-in page stays on WorkOS's own address for now (no custom domain).
F-009A link on the Appearance page relies on colour onlyFixedLinks inside text are now underlined everywhere.
F-010A scrolling strip on the Appearance page can't be reached by keyboardFixedThe sample table in the preview no longer scrolls.
F-011Links on the candidate join and instructions pages rely on colour onlyFixedSame fix as F-009.
F-01218 smaller accessibility issues (missing header area, heading order)FixedTop bar marked as the page header, the candidate join page has a page title, and heading levels on the marking page are in order. A re-scan is in the regression checklist (B43).
F-017No Coding Practice areaFixedLive at /practice (see scope item 1).
F-019No competitions, hackathons or leaderboardsFixedCompanies run competitions and hackathons on their own tests (Competitions in the menu): individuals or teams, an AI-drafted option that a person must approve, live standings, a copying check where a person clears or disqualifies each flagged entry, and a signed rank certificate with a badge and prize for every finisher. Public developer contests in the community get live standings and rank certificates too.
F-020AI Fluency, Certifications, SCIM user sync and the "agentic IDE" not seenFixedAI Fluency: a 0–100 rating in four parts (prompting, critique, verification, result) on every test that allows AI, replaced by the reviewer's marks once given and added to the skill profile and Passport. Certifications: exams on their own question pool, digitally signed certificates, renewal with a reminder. SCIM: directory sync for Okta, Entra ID and Google (Settings → Sign-in). Agentic IDE: a terminal and a live preview beside the coding agent, with every command recorded for the reviewer.
F-022Shareable test link doesn't show when it expiresFixedShows the closing date with the time left ("3 days left"), or "no closing date: it works until you turn it off".
F-025Question authors get a plain text box, not a code editorFixedThe candidates' code editor is used for answers, starter code and reference solutions.
F-026The code box is cramped on phonesFixedComes with F-025 (the editor sizes itself to the screen).
F-027Authors can't run a question against their own inputFixed"Test against my own input" added under the answer.
F-028A candidate's invitation link works only once; reopening it locks them outFixedLinks stay single-use (security), and a candidate whose link no longer works now types their email under "Email me a new link": a new link goes only to the address the company has, only while a test is open, with the same answer either way.
F-031"Notice period" and typing your full name are required before starting, with no explanationFixedA line under the button already says what is missing ("Type your name to start"). Which details are required is set by each company per test.
F-032The main address goes to sign-in; no public sign-upFixedThe main address now has public pages (home, pricing) with “Start free trial”, and self-service sign-up is live. Sign-in on Kombee’s own domain waits for the production WorkOS account (F-024).
F-033Should "View as Candidate" start a real interactive test?FixedIt opens a fresh sample candidate's real test (works now after the F-001 fix).
F-047Turning off a sign-in method may not sign out people already signed inFixedAnyone signed in that way is now signed out at once; the admin making the change stays in.
F-050AI Fluency thresholds not definedFixed75 and up Strong, 50–74 Developing, under 50 Needs support (KOMBEE-DECISIONS section 6).
F-051"Save for everyone" may stay disabled when only switching light/darkFixedThose buttons only choose which colours to edit (nothing to save yet). Renamed "Edit light colours" / "Edit dark colours" with a note saying so.
F-052Clearing the brand name falls back to the company code without explainingFixedThe hint now says what is used when it is left empty.
F-053API keys page doesn't show the API address or a docs linkFixedIt now shows both.
F-006Custom roles don't appear in "View as"DecidedDecided 12 Nov: No. "View as" is a demo tool and will be removed before real use (you asked to keep it until testing ends).
F-007"View as" for staff roles changes the view onlyDecidedDecided 12 Nov: Acceptable. Each staff role in "View as" signs in as a real sample user of that role, with that role's permissions. Only "Candidate" opens a separate site.
F-018Only 3 programming languages offered, not 20DecidedDecided: 12 languages at launch (Python, JavaScript, TypeScript, Java, C, C++, C#, Go, Rust, Kotlin, Ruby, PHP), more on request. New coding questions start with 7 ticked, each with starter code.
F-021API keys say "sandbox/live", the spec says "test/production"DecidedDecided 12 Nov: Keep "sandbox/live".
F-023No one-click download on the analytics number cardsDecidedDecided 12 Nov: No; table downloads are enough. Full downloads (spreadsheet and PDF) exist on the analytics tables.
F-024Sign-in goes to a WorkOS address without warningDecidedDecided 20 Nov: not for now. The production WorkOS sign-in page carries SetBenchmark's name and logo but stays on WorkOS's address; a custom sign-in domain costs about $99 a month. Revisit if buyers object.
F-030Tests only run in Chrome or Edge, so Firefox and Safari users are blockedDecidedDecided 12 Nov: Keep Chrome/Edge only, and say so up front. Done on purpose: monitoring and voice need features only Chrome and Edge have.
F-034"Try the editor first" practice before the timed testDecidedDecided 12 Nov: Keep. It reduces nerves and avoidable complaints.
F-035Rejecting and blocking a candidate needs two peopleDecidedDecided 12 Nov: On by default; a company may switch it off. Done on purpose (a fairness safeguard).
F-036Two timers (section and whole test) shown togetherDecidedDecided 12 Nov: Keep both. The section timer helps candidates pace each part; the overall timer shows the hard end.
F-037A− / A+ text size buttons for candidatesDecidedDecided 12 Nov: Keep.
F-038Names hidden while marking (blind marking) by defaultDecidedDecided 12 Nov: Keep. It reduces bias.
F-039Module names "Employee skills" and "Bench" vs "SkillUp" in the specDecidedDecided 12 Nov: Keep the screen names.
F-040AI policy labels differ from the specDecidedDecided 12 Nov: Keep "AI not allowed / allowed / required".
F-041"Find developers" marked Pilot, with an email introduction stepDecidedDecided 12 Nov: Yes, pilot at launch.
F-042Vendor portal shows agency proxy rates and a watch listDecidedDecided 12 Nov: Keep, worded as indicators to look into, not accusations; visible only to the company's own staff. This is scope item 2. Kombee's legal team should approve the wording shown about agencies.
F-043Bench sends reassessment invitations to employees automaticallyDecidedDecided 12 Nov: HR sends them deliberately; no second approver. Small change.
F-044Question Shield is not in the specDecidedDecided 12 Nov: Yes, add it. It protects the question bank.
F-048Which skill rating wins on the bench (manager, test or self-rating)?DecidedDecided 12 Nov: Tested score only; claimed skills show as "Stated" and count for nothing until tested. All three are shown.
F-049How question versions work isn't definedDecidedDecided 12 Nov: Published versions are fixed; edits make a new draft; tests keep the version they were built with. Each published version is fixed and tests use the version they were built with. Edits make a new draft.
F-054About 50 QA test candidates can't be deletedDecidedDecided: yes. 96 QA test records were erased on 12 Nov through the privacy service, with deletion certificates.
F-05616 places where the BRD and FRD disagreeDecidedDecided 12 Nov: Kombee issues v1.1 of the BRD and FRD; the corrections are drafted in KOMBEE-DECISIONS section 2.
F-003"Finish section" box may block clicks on the headerNo change neededChecked: it is a proper modal box, which blocks the page behind it on purpose until "Keep working", Esc, or "Finish section".
F-013Switching to Candidate signs the admin outNo change neededThe reviewers could not reproduce it.
F-014API addresses return "not found" on the websiteNo change neededWithdrawn by the reviewers: the API has its own address.
F-015Duplicate email warning when adding a candidateNo change neededWithdrawn by the reviewers.
F-016Old bug labels in a draft reportNo change neededThe reviewers' own paperwork. Nothing to change.
F-029"Add a candidate" is hidden inside a folded sectionNo change neededChecked: it is already a visible primary button that opens a panel; screen readers announce it as one.
F-046The on-screen timer pauses while the tab is hidden, then corrects itselfNo change neededThe server keeps the real time. The screen catches up when you return, and no time is lost or gained.
F-057The reviewers' own automated tests: 17 always pass and 17 check the wrong thingNo change neededThese are in the reviewers' test files, not in SetBenchmark. Their team should fix them. Our own test suite (2,114 tests) passes.
F-058A reviewer report quoted performance numbers that aren't in the specNo change neededThe reviewers’ own paperwork. Numeric targets have since been agreed (KOMBEE-DECISIONS section 3).