I need clarifications.

I see docker as a way to avoid having a standard dev platform for everyone in the company so that the infra team don't have to worry about patch xyz for library abc, only run docker.

But, with all the effort put in place to coordinate docker, k8s and all the shebang, isn't it finally easier to force a platform and let it slowly evolve over time?

Is docker another technical tool that tries to solve a non-technical problem?

Porque no los dos. Force a platform. Deploy non containerized. Build a dockerized version of the forced platform for cross-platform local dev.

I do not follow you. Every app has different needs. Containers encode them in a shareable way. You can evolve the image over time. So what more do you want?

Or supposedly you could shake a strainer in front of your eyes.

Still supposedly, the hardest part was finding the strainer in the kitchen without waking everyone in the house.

And the saddest part was discovering that it didn't work.

Well, in Quebec, the driver's insurance agency (SAAQ) decided to go with SAP and the major bosses were fired.

The cost of the migration was supposed to be 500millions $ and it's now estimated at 1.1 billion $.

But, they weren't fired because of SAP, they were fired because they lied to the government about the true cost.

Because it should be `today + 1 year + randomInt(1,42) days`.

Always include some randomness in test values.

Not a good idea for CI tests. It will just make things flaky and gum up your PR/release process. Randomness or any form of nondeterminism should be in a different set of fuzzing tests (if you must use an RNG, a deterministic one is fine for CI).

if it makes thing flaky

then it actually is a huge success

because it found a bug you overlooked in both impl. and tests

at least iff we speak about unit tests

Only if it becomes obvious why it is flaky. If it's just sometimes broken but really hard to reproduce then it just gets piled on to the background level of flakiness and never gets fixed.

To get around this, I have it log the relevant inputs, so it can be reproduced.

The whole concept of allowing a flaky unit test to exist is wild and dangerous to me. It makes a culture of ignoring real failures in what, should be, deterministic code.

Well, if people can't reproduce the failures, people won't fix them.

So, yes, logging the inputs is extremely important. So is minimizing any IO dependency in your tests.

But then that runs against another important rule, that integration tests should test the entire system, IO included. So, your error handling must always log very clearly the cause of any IO error it finds.

I remember having a flaky test with random number generation a few years ago - it failed very rarely (like once every few weeks) and when I finally got to fixing it, it was an actual issue (an off by one error).

This will often break on stuff like daylight saving changes, while almost as often you don't give a rats ass about the boundary behaviour.

That's why it's "randomInt(1,42)", not "randomLong()".

Generate fuzz tests using random values with a fixed seed, sure, but using random values in tests that run on CI seems like a recipe for hard-to-reproduce flaky builds unless you have really good logging.

> Always include some randomness in test values.

If this isn't a joke, I'd be very interested in the reasoning behind that statement, and whether or not there are some qualifications on when it applies.

humans are very good at overlooking edge cases, off by one errors etc.

so if you generate test data randomly you have a higher chance of "accidentally" running into overlooked edge cases

you could say there is a "adding more random -> cost" ladder like

- no randomness, no cost, nothing gained

- a bit of randomness, very small cost, very rarely beneficial (<- doable in unit tests)

- (limited) prop testing, high cost (test runs multiple times with many random values), decent chance to find incorrect edge cases (<- can be barely doable in unit tests, if limited enough, often feature gates as too expensive)

- (full) prop testing/fuzzing, very very high cost, very high chance incorrect edge cases are found IFF the domain isn't too large (<- a full test run might need days to complete)

I've learnt that if a test only fails sometimes, it can take a long time for somebody to actually investigate the cause,in the meantime it's written off as just another flaky test. If there really is a bug, it will probably surface sooner in production than it gets fixed.

Flaky tests are a very strong signal of a bug, somewhere. Problem is it's not always easy to tell if the bug's in the test or in the code under test. The developer who would rather re-run the test to make it pass than investigate probably thinks it's the test which is buggy.

sadly yes

people often take flaky test way less serious then they should

I had multiple bigger production issues which had been caught by tests >1 month before they happened in production, but where written off as flaky tests (ironically this was also not related to any random test data but more load/race condition related things which failed when too many tests which created full separate tenants for isolation happened to run at the same time).

And in some CI environments flaky test are too painful, so using "actual" random data isn't viable and a fixed seed has to be used on CI (that is if you can, because too much libs/tools/etc. do not allow that). At least for "merge approval" runs. That many CI systems suck badly the moment you project and team size isn't around the size of a toy project doesn't help either.

> it's written off as just another flaky test

So don't do that. That's bad practice. The test has failed for a reason and that needs to be handled.

Can't one get randomness and determinism at the same time? Randomly generate the data, but do so when building the test, not when running the test. This way something that fails will consistently fail, but you also have better chances of finding the missed edge cases that humans would overlook. Seeded randomness might also be great, as it is far cleaner to generate and expand/update/redo, but still deterministic when it comes time to debug an issue.

Most test frameworks I have seen that support non-determinism in some way print the random seed at the start of the run, and let you specify the seed when you run the tests yourself. It's a good practice for precisely the reasons you wrote.

Absolutely for things like (pseudo) random-number streams.

Some tests can be at the mercy of details that are hard to control, e.g. thread scheduling, thermal-based CPU throttling, or memory pressure from other activity on the system

There's another good reason that hasn't been detailed in the comments so far: expressing intent.

A test should communicate its reason for testing the subject, and when an input is generated or random, it clearly communicates that this test doesn't care about the specific _value_ of that input, it's focussed on something else.

This has other beneficial effects on test suites, especially as they change over the lifetime of their subjects:

* keeping test data isolated, avoiding coupling across tests * avoiding magic strings * and as mentioned in this thread, any "flakiness" is probably a signal of an edge-case that should be handled deterministically and * it's more fun [1]

[1] https://arxiv.org/pdf/2312.01680

Must be some Mandela effect about some TDD documentation I read a long time ago.

If you test math_add(1,2) and it returns 3, you don't know if the code does `return 3` or `return x+y`.

It seems I might need to revise my view.

I vaguely remember the same advice, it's pretty old. How you use the randomness is test specific, for example in math_add() it'd be something like:

  jitter = random(5)
  assertEqual(3 + jitter, math_add(1, 2 + jitter))

If it was math_multiply(), then adding the jitter would fail - that would have to be multiplied in.

Nowadays I think this would be done with fuzzing/constraint tests, where you define "this relation must hold true" in a more structured way so the framework can choose random values, test more at once, and give better failure messages.

> it's pretty old.

Damn, must be why only white hair is growing on my head now.

>Nowadays I think this would be done with fuzzing/constraint tests, where you define "this relation must hold true" in a more structured way so the framework can choose random values, test more at once, and give better failure messages.

So the concept of random is still there but expressed differently ? (= Am I partially right ?)

Yes, the randomness is still there but less manually specified by the developer. But also I haven't actually used it myself but had seen stuff on it before, so I had the wrong term: it's "property-based testing" you want to look for.

Here's an example with a python library: https://hypothesis.readthedocs.io/en/latest/tutorial/introdu...

The strategy "st.lists(st.integers())" generates a random list of integers that get passed into the test function.

And also this page says by default tests would be run (up to) 100 times: https://hypothesis.readthedocs.io/en/latest/tutorial/setting...

So I'm thinking... (not tested)

  @given(st.integers(), st.integers())
  def test_math_add(a, b):
      assert a + b == math_add(a, b)

...which is of course a little silly, but math_add() is a bit of a silly function anyway.

Randomness is useful if you expect your code to do the correct thing with some probability. You test lots of different samples and if they fail more than you expect then you should review the code. You wouldn't test dynamic random samples of add(x, y) because you wouldn't expect it to always return 3, but in this case it wouldn't hurt.

This sounds like the idea behind mutation testing

Interesting, haven't heard this before (I don't know much about testing). Is this kind of like fuzzing?

I recently had race condition that made tests randomly fail because one test created "data_1" and another test also created "data_1".

- Test 1 -> set data_1 with value 1

- Test 1 -> `do some magic`

- Test 1 -> assert value 1 + magic = expected value

- Test 2 -> set data_1 with value 2

But this can fail if `do some magic` is slow and Test 2 starts before Test 1 asserts.

So I can either stop parallelism, but in real life parallelism exists, or ensure that each test as random id, just like it would happen in real life.

Are you joking? This is the kind of thing that leads to flaky tests. I was always counseled against the use of randomness in my tests, unless we're talking generative testing like quickcheck.

or, maybe, there is something hugely wrong with your code, review pipeline or tests if adding randomness to unit test values makes your tests flaky and this is a good way to find it

or, maybe, it signals insufficient thought about the boundary conditions that should or shouldn't trigger test failures.

doing random things to hopefully get a failure is fine if there's an actual purpose to it, but putting random values all over the place in the hopes it reveals a problem in your CI pipeline or something seems like a real weak reason to do it.

I don't think anyone is advocating for random application of randomness.

`today` is random.

It's dynamic, but it certainly isn't random, considering it follows a consistent sequence

If "today" were random, our universe would be pretty fricken weird.

What is today right now in Australia? How about where you live? You have not thought enough about what you’re saying and are probably not aware of all the weird time issues we have in our world.

How?

In the 90's most software was closed source but cracks/trainer were always available.

Even for Rayman that had multiple (26?) cd-check during the game.

Security is mainly slowing the attacker because there's a maximum amount of stuff a human can do in 24hours. But now if you can simulate thousands of human attacking a system in different ways, it will crack.

Just like many stores have lock on their doors and, insurance if someone breaks the lock.

I'm guessing data security insurance will become a huge market in the years to come.

Aren't we in agreement then? Taking your lock analogy again, people don't put locks on their bikes because they protect them completely, but because they slow down someone who wants to steal them. Given enough resources everything will be cracked, it doesn't mean that making it harder is useless. People cracking games in the 90's may not have had the source code but they had the machine code and knew what to look for and where.

How would I do that?

Have "mycompany.com" and "marketing-mycompany.com"?

How

You've got it - subdomains are sufficient (IIRC). Something like notifications.mycompany.com for the important stuff and news.mycompany.com for the more marketing stuff (and of course someone.else.entirely for the the cold email list "well, surely this one little list couldn't hurt" ... nah, who am I kidding, never use those)

subdomains are not sufficient, unfortunately. I think the instance I'm thinking of is technically web/https (chrome's site safety feature where you can specify a domain as 3rd party content- so when something gets flagged uour entire domain doesn't get blacklisted)- but it's much easier to just grab another domain

I think the point is that subdomains are fine, assuming you ARE using a separate IP address for said subdomains.

The majority of my spam from established companies are on subdomains of their main domains. Are they doing it wrong?

Subdomains are fine. Newsletter.mycompany.com and application.mycompany.com or even mycompany.com.

Most of time, I would use mycompany.com for company stuff and mycompany.net for Application DNS and EMails.

Or the gun lobby isn't really happy that anyone can "print" a gun.

Yeah, no. I have shot 3D printed firearms with the head of the FPC, the most active gun lobby organization in the country.

Still better odds than winning the lottery right ? Right ?

Well, they heard that we don't like copilot in notepad so they removed "copilot" from notepad.

And right after that they added a brand new feature called tolipoc that will revolutionize the way you analyze your logs or modify your 17 year old cmd file!

Want to create a file with the current date and time? No need to google for it, tolipoc will do it for you!

Is the last sentence a reference to the .LOG classic notepad hack?

Must be some Mandela effect but I'm sure that Paint.net was supposed to replace mspaint when it was started.

Paint.NET wasn't Microsoft's, but was an independent app: https://en.wikipedia.org/wiki/Paint.NET

It was supposed to be a third-party replacement, sure, but certainly not an official one. It started as a student project. It's just the prefix that tricks your brain to associate it with MS's own .NET branded applications.

To be fair, the .NET brand is already super convoluted (there's .NET framework, the .NET core, .NET runtime, the .NET desktop runtime, the .NET sdk, and I'm genuinely not even sure which if any of these might refer to the same thing), on top of it weirdly sounding like something internet related to a casual user.

Yes, "Copilot" is not the first brand that MS has tried to stick to everything while being just as confused about it as (inevitably) the consumers. Although somehow they did manage to keep .NET mostly aimed at developers - besides the actual frameworks there's Visual Studio .NET and other dev tools, but I'm actually a bit surprised that they never had "Office .NET" or "Outlook .NET" or even "Windows .NET Edition" or something like that. Maybe they still had some sane people in charge of marketing and brand management back then.

They did brand the Microsoft accounts themselves, from “Passport” to “.NET Passport” for a while. That was before they were “Windows Live IDs.”