Skip to content

SparkException: There was a problem with creating a server #36

New issue
New issue
Open
Open
SparkException: There was a problem with creating a server#36
Assignees
tamersalamaondra-m
@tamersalama

Description

@tamersalama
tamersalama
opened on Nov 22, 2016

I'm getting org.apache.spark.SparkException: There was a problem with creating a server when attempting to call the rdd.take(..)

ruby 2.2.4p230
Windows 7
Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_25
Spark version 2.0.2
Scala version 2.11.8

Any pointers are appreciated

C:\path\to\test>ruby spark_test.rb
16/11/22 12:29:22 INFO spark.SparkContext: Running Spark version 1.5.0
16/11/22 12:29:22 INFO spark.SecurityManager: Changing view acls to: username
16/11/22 12:29:22 INFO spark.SecurityManager: Changing modify acls to: username
16/11/22 12:29:22 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(username); users with modify permissions: Set(username)
16/11/22 12:29:23 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/11/22 12:29:23 INFO Remoting: Starting remoting
16/11/22 12:29:23 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://[email protected]:57673]
16/11/22 12:29:23 INFO util.Utils: Successfully started service 'sparkDriver' on port 57673.
16/11/22 12:29:23 INFO spark.SparkEnv: Registering MapOutputTracker
16/11/22 12:29:23 INFO spark.SparkEnv: Registering BlockManagerMaster
16/11/22 12:29:23 INFO storage.DiskBlockManager: Created local directory at C:\path\to\username\AppData\Local\Temp\blockmgr-20f72b69-042d-4bde-8c0a-be0b5e544925
16/11/22 12:29:23 INFO storage.MemoryStore: MemoryStore started with capacity 1955.5 MB
16/11/22 12:29:23 INFO spark.HttpFileServer: HTTP File server directory is C:\path\to\username\AppData\Local\Temp\spark-860dc58a-60af-4551-9c21-2857427e9259\httpd-24c4f7a0-7cca-44e2-a5db-f3c779f0fa73
16/11/22 12:29:23 INFO spark.HttpServer: Starting HTTP Server
16/11/22 12:29:23 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/11/22 12:29:23 INFO server.AbstractConnector: Started [email protected]:57674
16/11/22 12:29:23 INFO util.Utils: Successfully started service 'HTTP file server' on port 57674.
16/11/22 12:29:23 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/11/22 12:29:23 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/11/22 12:29:23 INFO server.AbstractConnector: Started [email protected]:4040
16/11/22 12:29:23 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/11/22 12:29:23 INFO ui.SparkUI: Started SparkUI at http://10.73.14.60:4040
16/11/22 12:29:23 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/11/22 12:29:23 INFO executor.Executor: Starting executor ID driver on host localhost
16/11/22 12:29:23 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57693.
16/11/22 12:29:23 INFO netty.NettyBlockTransferService: Server created on 57693
16/11/22 12:29:23 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/11/22 12:29:23 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:57693 with 1955.5 MB RAM, BlockManagerId(driver, localhost, 57693)
16/11/22 12:29:23 INFO storage.BlockManagerMaster: Registered BlockManager
16/11/22 12:29:23 INFO spark.SparkContext: Added JAR /.ruby-spark.7c68fd15-69f6-4d3d-83b7-f34a4cbd7a06/ruby-spark.jar at http://10.73.14.60:57674/jars/ruby-spark.jar with timestamp 1479842963886
16/11/22 12:29:23 INFO Ruby: Ruby accumulator server is running on port 57694
16/11/22 12:29:24 WARN spark.SparkContext: sc.runJob with allowLocal=true is deprecated in Spark 1.5.0+
16/11/22 12:29:24 INFO spark.SparkContext: Starting job: Ruby
16/11/22 12:29:24 INFO scheduler.DAGScheduler: Got job 0 (Ruby) with 1 output partitions
16/11/22 12:29:24 INFO scheduler.DAGScheduler: Final stage: ResultStage 0(Ruby)
16/11/22 12:29:24 INFO scheduler.DAGScheduler: Parents of final stage: List()
16/11/22 12:29:24 INFO scheduler.DAGScheduler: Missing parents: List()
16/11/22 12:29:24 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (RubyRDD[1] at Ruby), which has no missing parents
16/11/22 12:29:24 INFO storage.MemoryStore: ensureFreeSpace(2632) called with curMem=0, maxMem=2050534932
16/11/22 12:29:24 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.6 KB, free 1955.5 MB)
16/11/22 12:29:24 INFO storage.MemoryStore: ensureFreeSpace(1664) called with curMem=2632, maxMem=2050534932
16/11/22 12:29:24 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1664.0 B, free 1955.5 MB)
16/11/22 12:29:24 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:57693 (size: 1664.0 B, free: 1955.5 MB)
16/11/22 12:29:24 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:861
16/11/22 12:29:24 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (RubyRDD[1] at Ruby)
16/11/22 12:29:24 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/11/22 12:29:24 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 2125 bytes)
16/11/22 12:29:24 INFO executor.Executor: Running task 0.0 in stage 0.0 (TID 0)
16/11/22 12:29:24 INFO executor.Executor: Fetching http://10.73.14.60:57674/jars/ruby-spark.jar with timestamp 1479842963886
16/11/22 12:29:24 INFO util.Utils: Fetching http://10.73.14.60:57674/jars/ruby-spark.jar to C:\path\to\username\AppData\Local\Temp\spark-860dc58a-60af-4551-9c21-2857427e9259\userFiles-1f4e6fa6-ebfa-41fd-9ebd-a7873e970559\fetchFileTemp3122839241910942098.tmp
16/11/22 12:29:24 INFO executor.Executor: Adding file:/C:/path/to/username/AppData/Local/Temp/spark-860dc58a-60af-4551-9c21-2857427e9259/userFiles-1f4e6fa6-ebfa-41fd-9ebd-a7873e970559/ruby-spark.jar to class loader
16/11/22 12:29:24 INFO ruby.FileCommand: New FileCommand at C:\path\to\username\AppData\Local\Temp\spark-860dc58a-60af-4551-9c21-2857427e9259\userFiles-1f4e6fa6-ebfa-41fd-9ebd-a7873e970559\command847668330604496435.cmd
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\path\to\test>16/11/22 12:29:34 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
org.apache.spark.SparkException: There was a problem with creating a server
        at org.apache.spark.api.ruby.RubyWorker$.createServer(RubyWorker.scala:100)
        at org.apache.spark.api.ruby.RubyWorker$.create(RubyWorker.scala:47)
        at org.apache.spark.api.ruby.RubyRDD.compute(RubyRDD.scala:50)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:88)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.spark.SparkException: Ruby master did not connect back in time
        at org.apache.spark.api.ruby.RubyWorker$.createMaster(RubyWorker.scala:161)
        at org.apache.spark.api.ruby.RubyWorker$.createServer(RubyWorker.scala:97)
        ... 10 more
Caused by: java.net.SocketTimeoutException: Accept timed out
        at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
        at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at org.apache.spark.api.ruby.RubyWorker$.createMaster(RubyWorker.scala:154)
        ... 11 more
16/11/22 12:29:34 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): org.apache.spark.SparkException: There was a problem with creating a server
        at org.apache.spark.api.ruby.RubyWorker$.createServer(RubyWorker.scala:100)
        at org.apache.spark.api.ruby.RubyWorker$.create(RubyWorker.scala:47)
        at org.apache.spark.api.ruby.RubyRDD.compute(RubyRDD.scala:50)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:88)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.spark.SparkException: Ruby master did not connect back in time
        at org.apache.spark.api.ruby.RubyWorker$.createMaster(RubyWorker.scala:161)
        at org.apache.spark.api.ruby.RubyWorker$.createServer(RubyWorker.scala:97)
        ... 10 more
Caused by: java.net.SocketTimeoutException: Accept timed out
        at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
        at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at org.apache.spark.api.ruby.RubyWorker$.createMaster(RubyWorker.scala:154)
        ... 11 more

16/11/22 12:29:34 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
16/11/22 12:29:34 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/11/22 12:29:34 INFO scheduler.TaskSchedulerImpl: Cancelling stage 0
16/11/22 12:29:34 INFO scheduler.DAGScheduler: ResultStage 0 (Ruby) failed in 10.583 s
16/11/22 12:29:34 INFO scheduler.DAGScheduler: Job 0 failed: Ruby, took 10.757870 s
16/11/22 12:29:34 INFO Ruby: Ruby accumulator server was stopped
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}
16/11/22 12:29:34 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}
16/11/22 12:29:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}
16/11/22 12:29:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}
16/11/22 12:29:35 INFO ui.SparkUI: Stopped Spark web UI at http://10.73.14.60:4040
16/11/22 12:29:35 INFO scheduler.DAGScheduler: Stopping DAGScheduler
16/11/22 12:29:35 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/11/22 12:29:35 INFO storage.MemoryStore: MemoryStore cleared
16/11/22 12:29:35 INFO storage.BlockManager: BlockManager stopped
16/11/22 12:29:35 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
16/11/22 12:29:35 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/11/22 12:29:35 INFO spark.SparkContext: Successfully stopped SparkContext
16/11/22 12:29:35 INFO Ruby: Workers were stopped
16/11/22 12:29:35 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/11/22 12:29:35 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
C:/Rubies/Ruby224/lib/ruby/gems/2.2.0/gems/ruby-spark-1.2.1/lib/spark/context.rb:306:in `method_missing': Job aborted due to stage failure: Task 0 in stage 0.0 fa16/11/22 12:29:35 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
 (SparkException)t recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): org.apache.spark.SparkException: There was a problem with creating a server
        at org.apache.spark.api.ruby.RubyWorker$.createServer(RubyWorker.scala:100)
        at org.apache.spark.api.ruby.RubyWorker$.create(RubyWorker.scala:47)
        at org.apache.spark.api.ruby.RubyRDD.compute(RubyRDD.scala:50)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:88)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.spark.SparkException: Ruby master did not connect back in time
        at org.apache.spark.api.ruby.RubyWorker$.createMaster(RubyWorker.scala:161)
        at org.apache.spark.api.ruby.RubyWorker$.createServer(RubyWorker.scala:97)
        ... 10 more
Caused by: java.net.SocketTimeoutException: Accept timed out
        at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)
        at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at org.apache.spark.api.ruby.RubyWorker$.createMaster(RubyWorker.scala:154)
        ... 11 more

Driver stacktrace:
        from C:/Rubies/Ruby224/lib/ruby/gems/2.2.0/gems/ruby-spark-1.2.1/lib/spark/context.rb:306:in `run_job_with_command'
        from C:/Rubies/Ruby224/lib/ruby/gems/2.2.0/gems/ruby-spark-1.2.1/lib/spark/rdd.rb:256:in `take'
        from spark_test.rb:27:in `<main>'

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions