Completed / CodeSOTA execution on White
Inspect the measurement.
Qwen/Qwen2.5-1.5B · AI2 ARC-Challenge (test) · 46.9% accuracy
32-example pilot. 32-example CodeSOTA declarative multiple-choice pilot. Not an official full-benchmark score.
Run ID / completed
293bd712648d308f4ffe2539
16 Oct 2026, 23:36 UTC
16 Oct 2026, 23:36 UTC
Model revision
8faed761d45a263340a0528343f099c05c9a4323Comparison scope
78d7cecafa6a6800233d22492171ba6acc9a8b644ca0ab6cc7013271bb6e4b06Evaluator SHA-256
4f6810ca652d8b80ae0c5e19f2a7e313e68357bb645a88a4d866e4c1141a0ed9Observed resources
3s elapsed · 2.93 GiB peak torch allocation
Dollar cost: not measured
Dollar cost: not measured
Download evidence
Example-level outputs
All 32 scored examples. Ground truth and model predictions are preserved in the downloadable JSONL.
{"choices":["Planetary density will decrease.","Planetary years will become longer.","Planetary days will become shorter.","Planetary gravity will become stronger."],"id":"0","log_likelihood_mean":[-3.9788193702697754,-4.2293524742126465,-3.3952219486236572,-3.8339321613311768],"prediction":2,"reference":2,"score":1.0}
{"choices":["buildings will be built faster","buildings will be made safer","building designs will look nicer","building materials will be cheaper"],"id":"1","log_likelihood_mean":[-3.4648149013519287,-3.3043906688690186,-4.4826884269714355,-3.5995419025421143],"prediction":1,"reference":1,"score":1.0}
{"choices":["Chemical energy is absorbed through the roots.","Light energy is converted to chemical energy.","Chlorophyll in the leaf captures light energy.","Sunlight is converted into chlorophyll."],"id":"2","log_likelihood_mean":[-3.793733596801758,-1.6590254306793213,-1.509004831314087,-2.7087464332580566],"prediction":2,"reference":2,"score":1.0}
{"choices":["the control","the hypothesis statement","the dependent (responding) variable","the independent (manipulated) variable"],"id":"3","log_likelihood_mean":[-6.188441276550293,-5.896124839782715,-3.528228998184204,-2.8766517639160156],"prediction":3,"reference":3,"score":1.0}
{"choices":["They have each lost kinetic energy.","They have each gained the same amount of potential energy.","They have each lost the same amount of potential energy.","They have each gained one-half of their maximum kinetic energy."],"id":"4","log_likelihood_mean":[-2.638762950897217,-1.6257693767547607,-1.5782979726791382,-1.9908031225204468],"prediction":2,"reference":3,"score":0.0}
{"choices":["a non-infectious, cell-cycle disease","an infectious, cell-cycle disease","a non-infectious, chronic disease","an infectious, chronic disease"],"id":"5","log_likelihood_mean":[-4.10479211807251,-5.840095043182373,-2.4373862743377686,-3.295917272567749],"prediction":2,"reference":1,"score":0.0}
{"choices":["to repare for migration before winter","to provide warmth during the cold winter months","to store food that will be eaten over the winter months","to protect the grasses and seeds from decay before winter"],"id":"6","log_likelihood_mean":[-5.3865227699279785,-2.0684690475463867,-2.105297803878784,-2.4099786281585693],"prediction":1,"reference":2,"score":0.0}
{"choices":["There was once more water in the area.","The area was once grassland.","The climate in the area was once tropical.","There are active faults in the area."],"id":"7","log_likelihood_mean":[-1.6850967407226562,-2.5917136669158936,-1.829853892326355,-3.0868337154388428],"prediction":0,"reference":2,"score":0.0}
{"choices":["The chicken population would go down.","Populations of mice and rats would increase.","Another bird of prey would replace the hawk.","The chickens would have a lower rate of disease."],"id":"8","log_likelihood_mean":[-2.2833571434020996,-2.7432501316070557,-2.290740489959717,-2.1862621307373047],"prediction":3,"reference":1,"score":0.0}
{"choices":["the atom","the electron","the nucleus","the proton"],"id":"9","log_likelihood_mean":[-2.0811195373535156,-4.120182037353516,-5.057682037353516,-5.456119537353516],"prediction":0,"reference":0,"score":1.0}
{"choices":["life processes","size differences","plasma membranes","energy molecules"],"id":"10","log_likelihood_mean":[-7.072242259979248,-7.130274772644043,-6.0237555503845215,-8.201766967773438],"prediction":2,"reference":1,"score":0.0}
{"choices":["The Sun influences the formation of waves.","The Sun creates water particles.","The Sun's rays cause organisms to come to the surface.","The Sun provides minerals."],"id":"11","log_likelihood_mean":[-2.8541147708892822,-4.125469207763672,-2.812986373901367,-5.1675944328308105],"prediction":2,"reference":0,"score":0.0}
{"choices":["32\u00b0 F","41\u00b0 F","78\u00b0 F","98\u00b0 F"],"id":"12","log_likelihood_mean":[-2.5324599742889404,-2.187089681625366,-2.154465436935425,-2.609321117401123],"prediction":2,"reference":2,"score":1.0}
{"choices":["Repeat the investigation.","Write a report of the results.","Make a table for recording data.","Make daily observations of the light bulbs."],"id":"13","log_likelihood_mean":[-4.8177690505981445,-2.7446868419647217,-3.193305492401123,-2.9649505615234375],"prediction":1,"reference":2,"score":0.0}
{"choices":["ozone","methane","water vapor","carbon dioxide"],"id":"14","log_likelihood_mean":[-5.541879177093506,-2.276254177093506,-1.7443022727966309,-0.499706894159317],"prediction":3,"reference":2,"score":0.0}
{"choices":["0 degrees Celsius","32 degrees Celsius","100 degrees Celsius","212 degrees Celsius"],"id":"15","log_likelihood_mean":[-0.7926574945449829,-1.5418568849563599,-1.6389707326889038,-2.286424160003662],"prediction":0,"reference":0,"score":1.0}
{"choices":["exclude research on teeth or bones","predict what the next discovery will be","analyze new data as it becomes available","delete earlier reports that were missing the new findings"],"id":"16","log_likelihood_mean":[-5.363903045654297,-2.764146089553833,-3.1494193077087402,-4.921311378479004],"prediction":1,"reference":2,"score":0.0}
{"choices":["from the ice to the tea","from the tea to the ice","from the pitcher to the tea","from the ice to the pitcher"],"id":"17","log_likelihood_mean":[-2.2988243103027344,-2.051400899887085,-1.9538906812667847,-2.9733035564422607],"prediction":2,"reference":1,"score":0.0}
{"choices":["the price of boards will increase","the price of boards will decrease","there will be more boards available","there will be more trees for logging"],"id":"18","log_likelihood_mean":[-2.277942657470703,-3.2961719036102295,-2.3787479400634766,-2.4281058311462402],"prediction":0,"reference":0,"score":1.0}
{"choices":["the discovery of the atom.","better surgical techniques.","continued experimentation.","the invention of the microscope."],"id":"19","log_likelihood_mean":[-2.788064956665039,-5.30203914642334,-4.616254806518555,-1.8343746662139893],"prediction":3,"reference":2,"score":0.0}
{"choices":["building proteins","breaking down wastes","controlling the activities of the cell","converting energy from one form into another"],"id":"20","log_likelihood_mean":[-6.511106491088867,-3.867898464202881,-3.1676082611083984,-3.346492290496826],"prediction":2,"reference":1,"score":0.0}
{"choices":["H_{2}O -> H + O + H","2H_{2}O(l) -> 2H_{2}(g) + O_{2}(g)","H:O:H -> H_{2}O","H_{2}O(l) -> 2H(g) + O(g)"],"id":"21","log_likelihood_mean":[-1.4395198822021484,-0.4085528552532196,-2.942112684249878,-1.0154436826705933],"prediction":1,"reference":1,"score":1.0}
{"choices":["Penguins can live in climates with freezing temperatures.","Penguins are fierce competitors.","Penguins are some of the most beautiful birds.","Penguins make great pets."],"id":"22","log_likelihood_mean":[-2.83280873298645,-4.334662914276123,-2.2975621223449707,-3.5259785652160645],"prediction":2,"reference":0,"score":0.0}
{"choices":["43\u00b0F.","50\u00b0F.","73\u00b0F.","90\u00b0F."],"id":"23","log_likelihood_mean":[-1.7772667407989502,-1.2385164499282837,-0.9642754793167114,-1.6576290130615234],"prediction":2,"reference":2,"score":1.0}
{"choices":["gas.","water.","wind.","clouds."],"id":"24","log_likelihood_mean":[-7.250013828277588,-4.70046329498291,-7.678035736083984,-5.8066816329956055],"prediction":1,"reference":1,"score":1.0}
{"choices":["It condensed.","It evaporated.","It became a gas.","It became a solid."],"id":"25","log_likelihood_mean":[-4.408642768859863,-3.0877275466918945,-3.7140462398529053,-2.339233160018921],"prediction":3,"reference":3,"score":1.0}
{"choices":["It is a constant climate feature throughout the year.","It is a seasonal weather feature with irregular occurrences.","It is a continuous weather feature during the season in which it occurs.","It is a rare event inconsistent with local climate and weather."],"id":"26","log_likelihood_mean":[-3.5398976802825928,-3.9106357097625732,-3.3550820350646973,-2.922313690185547],"prediction":3,"reference":1,"score":0.0}
{"choices":["Object 1, Object 2, Object 3","Object 1, Object 3, Object 2","Object 2, Object 1, Object 3","Object 2, Object 3, Object 1"],"id":"27","log_likelihood_mean":[-0.6669149398803711,-0.6722858548164368,-0.7045831680297852,-0.6960336565971375],"prediction":0,"reference":1,"score":0.0}
{"choices":["Plankton are colorful.","Plankton clean the water.","Plankton release oxygen.","Plankton reproduce quickly."],"id":"28","log_likelihood_mean":[-2.9295945167541504,-2.092060089111328,-2.233717918395996,-2.0589075088500977],"prediction":3,"reference":2,"score":0.0}
{"choices":["Wood is porous.","Wood is buoyant.","Wood is light.","Wood is magnetic."],"id":"29","log_likelihood_mean":[-3.8115811347961426,-2.8322560787200928,-3.809739589691162,-6.755402088165283],"prediction":1,"reference":1,"score":1.0}
{"choices":["how each organism died","the exact age of the fossils","when the fossils were discovered","the type of environment in which the organisms lived"],"id":"30","log_likelihood_mean":[-5.161923408508301,-2.2996976375579834,-2.9332313537597656,-1.884889006614685],"prediction":3,"reference":3,"score":1.0}
{"choices":["its ability to freeze","its specific heat capacity","its ability to exist as a liquid","its high surface tension"],"id":"31","log_likelihood_mean":[-2.805659294128418,-3.0963258743286133,-1.6146687269210815,-1.9714891910552979],"prediction":2,"reference":2,"score":1.0}