History Of Pain, What we learn about Traefik
This is just story about our journey from don’t know anything about the tool to actually use a tool. And how we end up hurt and fell pain debuging new tools that we don’t know so much about it, and switch back to the old and mature things that could full fill our desire. i’m not recommend you to choose the old one. Sometimes young and bold is better but sometimes its not. The best is understand your problem and use the right tools.
Note : All things works well when our team using nginx but we hungry for works so we search for alternatives even it means pain, kidding. actually we need reverse proxy that handle configuration easily and include forward auth (in this time we don’t know that nginx have auth_request module).
This story start when our beloved leader start random-ing about tools for our future platform and system. he is smart human with little bit hype driven development inside his mind, Sorry i’m kidding. Its base of our discussion and a lot of consideration we decide to use traefik for all our tools gateway. all dashboard and tools will go through traefik and serve it to backend.
I’m starting to try and taste traefik how it’s work and how easy the configuration is. you could combine traefik with its provider and one of the provider is key value consul or etcd. Thats really awesome. no need use consul-template like when we use nginx.
ok but first
What is Traefik
From traefik website, Traefik is a reverse proxy / load balancer that’s easy, dynamic, automatic, fast, full-featured, open source, production proven, provides metrics, and integrates with every major cluster technology. and they added No wonder it’s so popular! yay. for easy understand actually is nginx refound. with better in some way. All is good. traefik have nice dashboard
We could see even the health status how much http status thats comes in
In that time i said to myself, this is it! the tool that we waiting for in long time. nothing could satisfy us like traefik does.
In first phase our traefik is read configuration from etcd3. We hope we could change config easy and dynamically but it turns out that traefik and etcd3 is not match up. they dont get along that well. there is some the problem
- some times we should restrat traefik after change value in etcd.
- in some reason etcd is dead and lost of their keys in dashboard (its ui problem).
- we should deploy again with other prefix to make it works again.
- etcd ui not good enough so its hard to insert config.
for all above our purpose of using traefik and etcd not meet (could change config easy and dynamically)
after discuss again about the problem, we decide to using consul for key-value store traefik. This couple have good progress and chemistry, we could start seeing result.
- consul works well with traefik.
- traefik could read config from consul and make immediate change dynamically.
- we try out some backend and frontend. frontend rule, weight in url etc.
- we start to re route our subdomain to our traefik
There is some documentation about traefik and kv store here
In phase two i try to sore all config in consul, in phase one the global config still stored in file. so i use storeconfig feature to store file config to key-value store and restart traefik with
--consul --consul.endpoint=host:8500 .
and Boom its work smoothly.
Its works now and our jobs is done … … … . . for a while.
Problem Start Rampaging
After couple days. our team members asking about his streaming log is timeout and not fully loaded because is go through traefik. its ok when streaming directly to pods ip. I change something about idle timeout and its hard to test it, need a lot of work to just test it. so we leave it that way.
After 2 days the problem still exist and we found that actually traefik don’t support streaming if compressed true. (default http client in golang). and then problem and issue links start coming
- text/event-stream + compress=true not working https://github.com/containous/traefik/issues/2576
- some key in consul not parsed (see this code). its already merged PR for this bug but not released yet github pr (update september 2020 its already traefik 2.3 mate)
just those 2 problem make us wonder is traefik really good for us? and we start thinking to cheat from traefik and come back to our ex-tools nginx.
is she would be mad if we cheat on her?. we don’t really care because we have a dead line.
What we learn
- traefik is a great reverse proxy but not as mature as nginx.
- worth trying if we do not use event stream and flushing mechanism.
- traefik have good dynamic config with kv store no need extra work.
- don’t always trust yourself.
- new is not aways good. but old is not too. So, all tools is good for their field and specialization.
This writing is not about what is better, its about how we define our problem and match it with the tools available.
this is backup from my post on medium