So, you spend hours debugging, as we did some time ago, just to find out the “No Space Left On Device” error? It doesn’t matter what you do; the error keeps to pop out. If your disk is really full, then it is an easy problem to solve. Just clean it up. Hello, Provided attached file not showing clear error message. If there is disk space issue then you have contact to your service provided to check server disk space status as you have only cPanel Access level. 24x7serversolutions.com - Provide cPanel Server Management & Services.
- No Space Left On Device Error Failed
- Linux No Space Left On Device
- No Space Left On Device Error In Jenkins
- Cannot Write No Space Left On Device
I submitted an Apache Spark application to an Amazon EMR cluster. The application fails with a 'no space left on device' stage failure like this:
Freeing up space on your device may resolve the issue that you are seeing. You can do this by: 1) Remove any unused channels from your device. 2) Factory reset your device. AFAIK, two reasons may cause the 'No space left on device' error: (1) The physical disk has been used up. Run df -hk to check whether there is any free space left.
Short Description
Spark uses local disks on the core and task nodes to store intermediate data. If the disks run out of space, the job fails with a 'no space left on device' error. Use one of the following methods to resolve this error:
- Add more Amazon Elastic Block Store (Amazon EBS) capacity.
- Add more Spark partitions.
- Use a bootstrap action to dynamically scale up storage on the core and task nodes. For more information and a recommended bootstrap action script, see Dynamically scale up storage on Amazon EMR clusters.
Resolution
Add more EBS capacity
For new clusters: use larger EBS volumes
Launch an Amazon EMR cluster and choose an Amazon Elastic Compute Cloud (Amazon EC2) instance type with larger EBS volumes. For more information about the amount of storage and number of volumes allocated for each instance type, see Default EBS Storage for Instances.
For running clusters: add more EBS volumes
1. If larger EBS volumes don't resolve the problem, attach more EBS volumes to the core and task nodes.
2. Format and mount the attached volumes. Be sure to use the correct disk number (for example, /mnt1 or /mnt2 instead of /data).
3. Connect to the node using SSH.
4. Create a /mnt2/yarn directory, and thenset ownership of the directory to the YARN user:
5. Add the /mnt2/yarn directory inside the yarn.nodemanager.local-dirs property of /etc/hadoop/conf/yarn-site.xml. Example:
6. Restart the NodeManager service:
Add more Spark partitions
Depending on how many core and task nodes are in the cluster, consider increasing the number of Spark partitions. Use the following Scala code to add more Spark partitions:
No Space Left On Device Error Failed
Related Information
Linux No Space Left On Device
How can I troubleshoot stage failures in Spark jobs on Amazon EMR?
Anything we could improve?
Need more help?
Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.
To see how much storage you have available for Docker (dev/xvdcz), see AMI Storage Configuration and follow the instructions in To determine the available storage for Docker.
If you must extend the Amazon EBS volumes for your instances launched by AWS Batch, then increase Amazon EBS volumes in AWS Batch using a launch template.
If extending Amazon EBS volumes isn't the best solution for your scenario and you must go beyond the default storage limit of 10 GiB, then increase your Docker container storage volume limit for Amazon ECS instances launched from Amazon Linux 1 AMIs.
Note: You can use a launch template to build a configuration template that applies to all your Amazon Elastic Compute Cloud (Amazon EC2) instances launched by AWS Batch. A launch template allows you to customize the default Amazon ECS-optimized AMI used by AWS Batch without using a custom AMI.
No Space Left On Device Error In Jenkins
If you add user data to a launch template in the Amazon EC2 console, paste in the user data as plaintext, or upload the user data from a file.
Cannot Write No Space Left On Device
If you use the AWS CLI or an AWS SDK, you must first base64 encode the user data. Then, submit that string as the value of the UserData parameter when you call CreateLaunchTemplate. For example, see the following JSON template: