> we would ask you to use just your language’s standard library, no third-party libs or frameworks... With the problem description, we sent to candidates a functional test suite, a binary that when started would try to connect to the candidate’s server implementation, open lots of sockets, sends lots of messages, and verify the results against what the problem description stated. The candidate was instructed only to send their submission once it passed the functional test on their local box.
Incidentally, this is exactly how the "Sun Certified Java Developer" certification was structured. You were only allowed to use java.* and javax.* packages provided as part of the JRE, and even among those things like the SQL packages were off limits. Instead of sending a test suite, they provided an interface you needed to implement and then when you submitted it, they had some sort of test suite that tested your code for conformance to the spec (and if it failed, you failed). If it passed, someone would manually review your program to make sure it made sense with respect to the requirements (which were intentionally somewhat vague, like real requirements) and if you passed that step, you had to write an essay about WHY you made certain design decisions (like sockets+serialization vs. RMI, etc).
This was by far the most challenging and fun task of its type that I've done, and to this day if I see that someone has that certification I know that they have what it takes to get things done. This methodology should really be utilized more, although for many companies the overhead of designing such a challenge would be way over the top. Makes me think that things like https://www.stockfighter.io/ have a future in our industry.
Incidentally, this is exactly how the "Sun Certified Java Developer" certification was structured. You were only allowed to use java.* and javax.* packages provided as part of the JRE, and even among those things like the SQL packages were off limits. Instead of sending a test suite, they provided an interface you needed to implement and then when you submitted it, they had some sort of test suite that tested your code for conformance to the spec (and if it failed, you failed). If it passed, someone would manually review your program to make sure it made sense with respect to the requirements (which were intentionally somewhat vague, like real requirements) and if you passed that step, you had to write an essay about WHY you made certain design decisions (like sockets+serialization vs. RMI, etc).
This was by far the most challenging and fun task of its type that I've done, and to this day if I see that someone has that certification I know that they have what it takes to get things done. This methodology should really be utilized more, although for many companies the overhead of designing such a challenge would be way over the top. Makes me think that things like https://www.stockfighter.io/ have a future in our industry.