Ugh. certifi the most annoying thing ever. It snuck into numerous python dependencies, never made anything better, and confuses all the junior devs when their venvs/containers etc can't access internal CA signed resources. Probably have explained what to do over 9,000 times. And for what? "To provide updated root CAs". Meh. I don't need daily CA trust store updates. If your CA roots are that new, I want nothing to do with it. Cert issuers (cough sectigo and their amateur hour CA root rotation recently, not even Mozilla had it).
I remember when a group of friends went to see Rocky Horror at Clinton Street theater here in Portland, not having any idea what we were getting ourselves into. What a memory. Rest in peace Tim Curry
When was that? Just went there last year - they technically never stopped the Saturday night shows during the pandemic, so we’d both have watched the same uninterrupted theatrical run…
Did similar, but went 12x16, trenched power and cat6, did rockwool insulation, did pine tongue and groove paneling and an Alaskan white cedar porch.
When we sold the house, I'm pretty sure the shed is what sealed the deal, people had all sorts of things they wanted it to be, even as a spare bedroom
i did a 24x36, with a loft. 2015, 32,000 bucks from lavalley's in newport nh, installed 80 miles away in another random town. poured slab. 6 windows, 2 garage doors. i realized, all i did was build a garage, not a work from home office.
but over the next few years i learned so many hobby art projects: insulation, electrical, sheetrock. a few inspections, a few consults. a few surprises that people in this state can just go buy the stuff at a box store!
and then eventually, my 12x6 foot office appeared, the same width as a 1960s metal tanker desk. finally!
and ... i instantly hated working out there. i didnt make it 6 weeks. it's just an office for storing crap, now: i work out of the house, where the kitchen, dogs, bathroom, are. but the journey was worth it, even if the destination was not.
I disagree. I work in a large environment, and rsyslog is where 90% of data goes to first. It can keep up with millions of messages per second, route them to higher order services for indexing (bigquery, splunk, elastic etc etc). Has rules engines, encryption, supports multiple protocols and obviously has TLS too. You can surely augment with otel and such where you can, but syslog is uhhhh, deployed in so many places that it would make an average app developer's head spin when all they're used to is application logging in a controlled structured place in their silo.
On the other hand, traditional syslog is UDP based, so as soon as the receiver experiences CPU or I/O starvation and its receive buffer overflows, it will begin dropping messages. That's not great for observability, and may well be impermissible at many sites that need end-to-end log integrity (e.g. audit logs).
cheaply dropping log msgs you cannot handle is absolutely essential for an observability system - otherwise excess load can take down the logging infra which can (if msgs aren't dropped) take down the prod network/app trying to send reliable log msgs.
That's fine if you plan for it and can clearly delineate which logs can be dropped and which can't. The challenge is that most applications log to a single stream that consists of both high and low-priority logs muxed together and is sent to a single destination, making it impossible to distinguish the two, and there's only one receive buffer.
Excess load can't take down a well-engineered log collection infrastructure. There can be overload, but the backpressure should propagate downstream and senders and intermediaries should buffer locally if needed. Once the collectors are able to catch up again, the spooled messages will be dispatched, and the backlog should recover.
A well-engineered logging system for sites that care about integrity and durability should look a lot like a distributed message queue.
> Audit logs are a distinct feature.
In my experience, this is not always as distinct as one might hope. On multiple occasions in my career, a customer demanded we perform research using our logs to answer, and the information they sought were not in the class of logs that were considered "audit logs" in advance. Everyone chooses differently what qualifies as "audit logs"; it doesn't have an objective definition.
If you are doing massive log streams with mixed priority/durability, use named queues if you have different priority and retry and durability needs. You keep bringing up edge cases, but they've already been considered and solutions already engineered and available in every major syslog implementation.
I didn't say they those choices aren't viable; I'm strictly talking about architectural decisions. You can solve the problem with different solutions, be they rsyslog or otherwise. That said, I probably wouldn't go with rsyslog as my default choice anymore since the world is moving on to OpenTelemetry.
> There can be overload, but the backpressure should propagate downstream and senders and intermediaries should buffer locally if needed.
You're only delaying the inevitable. Even with local buffering, you can arrive at a point where you can buffer no more and have either to choke the production workload or start dropping messages.
I wouldn't call that "inevitable." Buffering exists for a reason--to buy you a bit of safety for short-lived busy periods (or, in the case of queues, delayed or temporarily failed consumers). It also amortizes the cost of I/O. Buffering isn't just for logging; it's all over the networking and I/O stacks.
This was a solved problem a long time ago in rsyslog. One can define a local spool and enable TCP (and optionally encryption) to multiple syslog servers. If something interrupts the flow the syslog messages will queue locally and then de-spool when communications are restored.
Thirty years ago and more that might have been a valid objection, but at that time the alternatives weren't great either. No one has suggested running syslog over unreliable transport after that.
In fact, the queue management and at least the possibility of some rudimentary end-to-end cryptographic integrity checks are some of the stronger points of rsyslog. Splunk Cloud and Elastic, as far as I know, lacks the latter completely which rules them out as a single log sink for environments with that type of requirements.
Which major Linux distro ships rsyslog with TCP as the default remote protocol and durable local-buffer configuration out of the box for remote delivery? Genuinely curious.
What Linux distro ships with remote logging out of the box at all? Hopefully none, because that would be be absurd. To whom would those logs be sent?
That documentation link probably isn't as telling as is suggested, because the next example is for tcp. In the old days before tcp support was widespread (looking at you, Java) it was common to listen for udp on localhost so it was probably a common configuration.
There's not much to debate here. Syslog is used everywhere and the main reasoosn are that it is very reliable, trivial to load balance, and popular implementations have integrity checking that is permissible in regulatory environments. You can criticize it for many things, for example that most parsers are much too liberal or that the facility and severity fields are clearly dated, but not for being unreliable.
"Beginner tutorial", I'm sorry for not taking your point seriously, but I can't take it seriously. TCP for syslog (and RELP) have been around a long time (late 90s for syslog over TCP, 2006 for RELP).
rsyslog and syslog-ng support it all, and operators have had choices given the import of their log data and what they can tolerate.
> "Beginner tutorial", I'm sorry for not taking your point seriously, but I can't take it seriously.
Well, maybe go observe how a broad array of sites implement it in practice, then you might take it more seriously. Maybe you don't implement it that way, but a lot of people will just follow the tutorials or shortcut their way to something that works (but is brittle).
At any rate, I was responding directly to the claim that "No one has suggested running syslog over unreliable transport" which is obviously untrue.
Yes, go observe a broad array of sites - for someone who says theyre a (non)practicing attorney, you know in environments where logs and audit are considered evidence, to such a degree that they must be reliably transported and immunutable, someone doesn't just turn on UDP syslog to a box and let it sit there. Architecture and implementation happen, where it matters. So what if anyone uses otel or syslog, people can configure em both to be lossy or lossless, I struggle to understand the "gotchas" you point to.
Then I don't know what to tell you. *shrug* I feel like you're arguing just for argument's sake, and I'm not really interested in having a conversation with someone who's not demonstrating open-mindedness or a willingness to learn from others' experiences in the field.
You can use TCP and other mechanisms for guaranteed delivery. Like, almost all LIDR logging across the planet uses TCP syslog, which is durable and attestable to in courts.
> TCP syslog...which is durable and attestable to in courts.
TCP alone won't get you there. It's certainly not durable in and of itself. All TCP can do is ensure that streamed data is received in the correct order, and confirm that a segment's data was successfully placed into the right buffer on the receiver side. You also need immutable storage, stronger integrity checks than what TCP itself provides, and many other requirements I haven't researched in a while.
As someone whose built megabyte to petabyte scale system of records, "yah but" to nits is annoying. You've solved it all yourself in your thought exercise though. Saying "the world has moved on", eh, not in any sense of the legal world, no. There's entire ecosystems around syslog alone to satisfy everything, otel is a baby fart feature and reliability and attestable wise
There’s no need to be rude and dismissive. Knock it off. You’re coming across as unnecessarily provocative and defensive.
If you truly believe OTel is a “baby fart,” I’d recommend you go to SRECon, KubeCon, and other similar conferences and make your opinion loudly known. Nobody talks about syslog there, and these are big and well-respected companies represented there.
Syslog is mature and battle tested for what it is: schlepping unstructured plaintext logs from one place to another. It’s not really purpose-built to meet the needs of a full-fledged observability solution, though, of which logs are but a component.
syslog isn't an observability system, you're right. It's just a transport. It doesn't really care what it transports either, delimeted, positional, structured - it'll transport it all. It'll do it durably, accurately and reliable with very modest systems at millions of messages a second.
The goal is to get it to the right place for the observability systems, and to feed them reliably.
We keep local logs for 24 hours, a sideline feed to S3, they then get sent to multiple multiple systems for tracing and legal and audits and siem etc etc.
It's not trying to be a foglight or datadog, it just helps enable them.
Apologies for being argumentative, but I come into orgs who make these generalizations, and I discover (or they engage me) because they've missed critical logging paths and face legal and compliance or intrusions and want to build more robust logging.
Otel is certainly part of it, but writing off syslog because of configuration options that aren't durable etc, when that hasn't been the baseline in decades - well, it keeps me employed at a high salary because I see this often
And the number of k8s envs that log stdout through them into.... more rsyslog, it's truly everywhere. Plus all the sidecar containers deployed that shuffle app logs, lots of syslog there, its so lightweight and simple and reliable. I watch all the gyrations people go through to achieve the same result, and it's always changing, hurts my brain thinking how much time they waste
In our case, other than network switches and routers, Syslog is the log of record and Splunk runs on all the VMs. Splunk isn’t forwarding syslog itself
This is going to leave a mark for centuries, history books are going to look back on this period in the US in complete dismay and horror for what this country continues to do.
As an American, and from my point of view, this is a headline moment where this country commits to its slide to just a middling country with "only" 380 million people mismanaged to inefficiency and un competitiveness and brain drain. The whole rest of the world has their own agency and is wising up to the fact that the US is limited to financial instruments and 5-13 functional aircraft carriers to project force. Soft power and goodwill is spent and we're just ignorant bullies who elected a rapist twice and can't be trusted
> The whole rest of the world has their own agency and is wising up to the fact that the US is limited to financial instruments and 5-13 functional aircraft carriers to project force. Soft power and goodwill is spent and we're just ignorant bullies who elected a rapist twice and can't be trusted
Maybe it's just the American way of passing the torch, this is kind of a gift by reminding the rest of the world about what they can do without us. The "America, defender of democracy" image was an illusion constructed as we emerged from WW2 as the strongest of the Allies, and that of course was due to the unique privilege of deciding when to tip the scales strategically, not when we were forced to.
So much has been built on that American privilege at all levels since then. The most glaringly obvious folly was interpreting this privilege as American Exceptionalism, because as a "leader of the free world" obviously that depends on how the rest of the world feels about it.
Did you miss the 1970s and the impact of Vietnam (that lasted through to the 90s for that generation)? The malaise about American from the 70s through the 80s? The corporate raiders of the 80s and the destruction of so many Americans lives in that period? The rust belt? Many many Americans of older generations losing their pensions because of Wall Street firms bs destruction of companies? The brain drain from the majority of the US into a few metro areas?
These are all things huge parts of the USA has dealt with for generations at this point.
Why do rust belt and brain drain area supposedly hsrmed by all that keep voting for pro wall street politicians and politics? They also keep voting pro-wars.
Well the current president was very anti-war in his campaign, also wanted to replace Obama care with "something better". I'm not saying the voting population in those areas is stupid, just very gullible.
reply