Repository navigation
fix(database): drop programs rejected by the novelty check - #528
Open
flyingdoubleG wants to merge 1 commit into
Open
flyingdoubleG wants to merge 1 commit into
flyingdoubleG wants to merge 1 commit into
Conversation
_add() stored the candidate in self.programs before the novelty check and returned its id when the check failed, leaving the rejected program in the database: it could be sampled as a parent, ranked by get_top_programs(), counted against population_size and written to checkpoints. Callers could not tell it was rejected, so the parallel controller stored its artifacts, traced it and bumped the wrong island's generation counter. Remove the provisional entry and return None, matching the existing admission-strategy rejection contract, and compute feature coordinates after the novelty check so rejected programs do not widen feature scaling.
|
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ProgramDatabase._add()stores the candidate inself.programsbefore running the novelty check (the check looks the program up by id), but when the check fails it returnsprogram.idwithout undoing that store:So a program the judge rejected as NOT_NOVEL is still in the database. It is not on any island, but it
_sample_random_parent()(and therefore bysample()),get_top_programs()/get_best_program(metric=...),population_size—_enforce_population_limit()is skipped on this path, so the database can grow past the limit,save(),_calculate_feature_coords().Since
add()returns the id, callers can't tell it was rejected either:ProcessParallelControllerstores its artifacts, logs it to the evolution trace as an accepted child and increments the generation counter ofcurrent_island(the rejected program has noislandmetadata) instead of the target island.This path only became reachable after #483 fixed the NOT_NOVEL parsing; before that the judge never rejected anything.
Minimal repro (
population_size=3, one island, identical embeddings, judge always answersNOT_NOVEL):Fix
None. This is the contractadd()already has forpopulation_strategy.admitrejections, and both existing callers handle it (ProcessParallelControllerraises_CandidateRejected,_migrate_onechecksis not None).feature_stats.Tests
New
tests/test_novelty_rejection.py(mocked embedding client + a judge that always returnsNOT_NOVEL): rejected programs are not stored, never sampled or ranked, not checkpointed, and don't touchfeature_stats. All 4 fail onmainand pass with this change. Existing novelty / migration / population-strategy tests still pass; the rest of the unit suite has the same results asmainlocally.