4.25 silently lowers vm.max_map_count from 262144 to 65530, breaking Elasticsearch containers ("ExitCode": 78)#7047

jamshid · 2023-10-27T20:42:29Z

Description

Elasticsearch containers rely on vm.max_map_count being 262144
https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
and that was the default for Docker for Mac up until 4.25.

Reproduce

docker run centos:7 sysctl vm.max_map_count on old and new Docker for Mac. Now it outputs:
vm.max_map_count = 65530
Previously it output:
vm.max_map_count = 262144

Expected behavior

Upgrades shouldn't break existing deployments or lower sysctl values.

docker version

Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:28:49 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.25.0 (126437)
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:36 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.6
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.5
    Path:     /Users/jamshid/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.0-desktop.1
    Path:     /Users/jamshid/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/jamshid/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/jamshid/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.9
    Path:     /Users/jamshid/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/jamshid/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/jamshid/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.0.9
    Path:     /Users/jamshid/.docker/cli-plugins/docker-scout

Server:
 Containers: 58
  Running: 23
  Paused: 0
  Stopped: 35
 Images: 364
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 6.4.16-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 15.61GiB
 Name: linuxkit-7ad88f20e5c4
 ID: c6944eec-13eb-4b2b-96b3-beb3e4774da1
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  192.168.1.50:5100
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

76189D97-5C98-4A66-A89B-45AC1DEF5F42/20231027203626

Additional Info

No response

The text was updated successfully, but these errors were encountered:

jamshid · 2023-10-27T20:50:24Z

As a temporary workaround you can run this but idk how to make it permanent on macOS.
docker run --rm -ti --privileged centos:7 sysctl vm.max_map_count=262144

I think there is now a way to permanently adjust sysctl values with Docker for Windows but don't know it offhand.

On linux of course it's just this on the host:
echo 'vm.max_map_count = 262144' > /etc/sysctl.d/98-elasticsearch.conf

jamshid · 2023-10-27T21:00:11Z

Finally, I'm not sure this is related to this problem but I think this s a new error in 4.25 with my elasticsearch container:
docker-compose logs -f elasticsearch shows a lot of expected logs then quits mysteriously with:
error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

thomet · 2023-11-01T11:59:26Z

I'm experiencing the same problem on 4.25, quite easy to reproduce it:

❯ docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

❯ docker run -it --privileged alpine:latest
/ # sysctl vm.max_map_count
vm.max_map_count = 65530
/ # sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144
/ # sysctl vm.max_map_count
vm.max_map_count = 262144
/ # exit

❯ docker ps -a
CONTAINER ID   IMAGE           COMMAND     CREATED          STATUS                      PORTS     NAMES
ff0d3a4df7b4   alpine:latest   "/bin/sh"   24 seconds ago   Exited (0) 11 seconds ago             strange_hamilton

❯ docker start -ai strange_hamilton
/ # sysctl vm.max_map_count
vm.max_map_count = 262144
/ # exit

❯ echo 'Now I restart docker-desktop'
Now I restart docker-desktop

❯ docker start -ai strange_hamilton
/ # sysctl vm.max_map_count
vm.max_map_count = 65530

diepes · 2023-11-02T11:41:50Z

Just logged a support ticket for elastisearch dying with error.

2023-11-03 00:07:11 ERROR: [1] bootstrap checks failed
2023-11-03 00:07:11 [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

desmortum · 2023-11-02T14:57:42Z

Same here, reproduced on two machines. Couldn't remedy easily. Rolled back.

plessbd · 2023-11-03T00:59:25Z

As a temporary workaround you can run this but idk how to make it permanent on macOS. docker run --rm -ti --privileged centos:7 sysctl vm.max_map_count=262144

I think there is now a way to permanently adjust sysctl values with Docker for Windows but don't know it offhand.

On linux of course it's just this on the host: echo 'vm.max_map_count = 262144' > /etc/sysctl.d/98-elasticsearch.conf

Looks like there is currently no way to make the changes perm

"the filesystem of Docker Desktop is readonly and you can’t write or install anything in it"

https://forums.docker.com/t/how-to-enter-docker-vm-terminal-on-macos-14/138316

jamshid · 2023-11-03T05:29:45Z

FWIW this workaround for compose users came up in:
https://dockercommunity.slack.com/archives/C2TJ8FEJY/p1698948708883499?thread_ts=1698438548.692899&cid=C2TJ8FEJY

Add a privileged compose service, make the elasticsearch service depends_on it.

  fixsysctl:
    restart: "no"
    privileged: true
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 25m
    image: centos:7
    command: "sysctl -w vm.max_map_count=262144"

haloboy777 · 2023-11-03T09:17:36Z

Faced the same issue.

Like @desmortum said, Rolling back to v4.24.2 fixes the issue.

Release note link: https://docs.docker.com/desktop/release-notes/#4242

Deadvi5 · 2023-11-06T09:09:12Z

Same here, in my team we're all affected by this

chdimitrov · 2023-11-07T17:24:25Z

We are affected by this too.
We managed to fix it for our need following this steps
but this is not a persistent solution if you restart the docker VM you have to apply it again
log in to the docker VM via:  
nc -U ~/Library/Containers/com.docker.docker/Data/debug-shell.sock
than execute:
sysctl -w vm.max_map_count=262144

jamshid · 2023-11-07T19:59:26Z

@chdimitrov cool but as mentioned in #7047 (comment) you can also simply run a privileged container to do it:

docker run --rm -ti --privileged centos:7 sysctl vm.max_map_count=262144

This can also be made a compose service to have it automatically run before elasticsearch container startup, see above.

mciricean · 2023-11-07T20:52:28Z

same here, went to prior version until this is fixed https://docs.docker.com/desktop/release-notes/#4242

Tostti · 2023-11-08T11:15:55Z

Same problem here on Manjaro Linux.

FWIW this workaround for compose users came up in: https://dockercommunity.slack.com/archives/C2TJ8FEJY/p1698948708883499?thread_ts=1698438548.692899&cid=C2TJ8FEJY

Add a privileged compose service, make the elasticsearch service depends_on it.
  fixsysctl:
    restart: "no"
    privileged: true
    deploy:
      replicas: 1
      resources:
        limits:
          memory: 25m
    image: centos:7
    command: "sysctl -w vm.max_map_count=262144"

This workaround worked perfect for me. Thanks!

lorenrh · 2023-11-14T10:10:43Z

We've released a new version of Docker Desktop containing a fix for this issue, you can see the release notes! I'll close this issue for now, but if the issue persists, feel free to open a new issue!

jamshid added the needs-triage label Oct 27, 2023

docker-robot bot added the version/4.25.0 label Oct 27, 2023

bsousaa added status/triage and removed needs-triage labels Nov 2, 2023

lorenrh added status/acknowledged and removed status/triage labels Nov 6, 2023

lorenrh closed this as completed Nov 14, 2023

dividor mentioned this issue Aug 18, 2024

[Subtask]: update website documentation install guide infiniflow/ragflow#1919

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

4.25 silently lowers vm.max_map_count from 262144 to 65530, breaking Elasticsearch containers ("ExitCode": 78)#7047

4.25 silently lowers vm.max_map_count from 262144 to 65530, breaking Elasticsearch containers ("ExitCode": 78) #7047

jamshid commented Oct 27, 2023

jamshid commented Oct 27, 2023

jamshid commented Oct 27, 2023 •
edited

Loading

thomet commented Nov 1, 2023 •
edited

Loading

diepes commented Nov 2, 2023

desmortum commented Nov 2, 2023

plessbd commented Nov 3, 2023

jamshid commented Nov 3, 2023

haloboy777 commented Nov 3, 2023

Deadvi5 commented Nov 6, 2023

chdimitrov commented Nov 7, 2023

jamshid commented Nov 7, 2023

mciricean commented Nov 7, 2023

Tostti commented Nov 8, 2023

lorenrh commented Nov 14, 2023

4.25 silently lowers vm.max_map_count from 262144 to 65530, breaking Elasticsearch containers ("ExitCode": 78)#7047

4.25 silently lowers vm.max_map_count from 262144 to 65530, breaking Elasticsearch containers ("ExitCode": 78) #7047

Comments

jamshid commented Oct 27, 2023

Description

Reproduce

Expected behavior

docker version

docker info

Diagnostics ID

Additional Info

jamshid commented Oct 27, 2023

jamshid commented Oct 27, 2023 • edited Loading

thomet commented Nov 1, 2023 • edited Loading

diepes commented Nov 2, 2023

desmortum commented Nov 2, 2023

plessbd commented Nov 3, 2023

jamshid commented Nov 3, 2023

haloboy777 commented Nov 3, 2023

Deadvi5 commented Nov 6, 2023

chdimitrov commented Nov 7, 2023

jamshid commented Nov 7, 2023

mciricean commented Nov 7, 2023

Tostti commented Nov 8, 2023

lorenrh commented Nov 14, 2023

jamshid commented Oct 27, 2023 •
edited

Loading

thomet commented Nov 1, 2023 •
edited

Loading