Closed
Description
Issue Template
Title: Envoy crashed when click config_dump from admin page
Description:
I started envoy with xds server, after it bootstrapped, then clicked config_dump on admin page, it generates following stack trace.
Version:
envoy version: 08712e9/1.7.0-dev/Modified/DEBUG
- Config *:
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address: { address: 127.0.0.1, port_value: 9902 }
dynamic_resources:
cds_config:
api_config_source:
api_type: GRPC
cluster_names: [xds_cluster]
lds_config:
api_config_source:
api_type: GRPC
cluster_names: [xds_cluster]
static_resources:
clusters:
- name: xds_cluster
connect_timeout: 0.25s
type: STATIC
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
hosts: [
{ socket_address: { address: 10.1.62.43, port_value: 8083 }}
]- Stack trace *
[2018-05-09 11:50:07.205][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:114] Caught Segmentation fault: 11, suspect faulting address 0x0
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:87] Backtrace obj thr<0>:
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj<envoy 0x000000010ba08026 _ZN5Envoy6Router30RouteConfigProviderManagerImpl29getStaticRouteConfigProvidersEv>
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> #0 0x10ba08026:
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:99] thr<0> obj
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Initial import #1 0x10ba0b906: Envoy::Router::RouteConfigProviderManagerImpl::dumpRouteConfigs() + 1222
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Initial import #2 0x10ba1ba0e: Envoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3::operator()() const + 30
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> network filter: fix upstream host storage #3 0x10ba1b9e3: std::__1::__invoke_void_return_wrapper<std::__1::unique_ptr<google::protobuf::Message, std::__1::default_deletegoogle::protobuf::Message > >::__callEnvoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3&(std::__1::unique_ptr<google::protobuf::Message, std::__1::default_deletegoogle::protobuf::Message >, Envoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3&&&) + 51
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Failure reason desc #4 0x10ba1b89f: std::__1::__function::__func<Envoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3, std::__1::allocatorEnvoy::Router::RouteConfigProviderManagerImpl::RouteConfigProviderManagerImpl(Envoy::Server::Admin&)::$_3, std::__1::unique_ptr<google::protobuf::Message, std::__1::default_deletegoogle::protobuf::Message > ()>::operator()() + 47
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> More details around failure reasons. #5 0x10be88bf9: std::__1::function<std::__1::unique_ptr<google::protobuf::Message, std::__1::default_deletegoogle::protobuf::Message > ()>::operator()() const + 137
[2018-05-09 11:50:07.228][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> fix filter manager crash #6 0x10be885e2: Envoy::Server::AdminImpl::handlerConfigDump(absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) const + 898
[2018-05-09 11:50:07.229][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> read --build-id from root cmake project during linking #7 0x10bec2df7: Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4::operator()(absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) const + 71
[2018-05-09 11:50:07.229][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: build/install and misc #8 0x10bec2da2: std::__1::__invoke_void_return_wrapperEnvoy::Http::Code::__call<Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4&, absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&>(Envoy::Http::Code, Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4&&&, absl::string_view&&, Envoy::Http::HeaderMap&&&, Envoy::Buffer::Instance&&&) + 178
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> add x-envoy-upstream-rq-per-try-timeout-ms router header option #9 0x10bec2c38: std::__1::__function::__func<Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4, std::__1::allocator<Envoy::Server::AdminImpl::AdminImpl(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::Server::Instance&, std::__1::unique_ptr<Envoy::Stats::Scope, std::__1::default_deleteEnvoy::Stats::Scope >&&)::$_4>, Envoy::Http::Code (absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&)>::operator()(absl::string_view&&, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) + 88
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Add stats for timeouts #10 0x10bea2774: std::__1::function<Envoy::Http::Code (absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&)>::operator()(absl::string_view, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) const + 196
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> ci: do asan build #11 0x10be9cffa: Envoy::Server::AdminImpl::runCallback(absl::string_view, Envoy::Http::HeaderMap const&, Envoy::Http::HeaderMap&, Envoy::Buffer::Instance&) + 1066
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Refactor async client #12 0x10be82f55: Envoy::Server::AdminFilter::onComplete() + 1013
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: deployment types #13 0x10be82b49: Envoy::Server::AdminFilter::decodeHeaders(Envoy::Http::HeaderMap&, bool) + 57
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: admin #14 0x10bf05959: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter*, Envoy::Http::HeaderMap&, bool) + 1657
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: misc #15 0x10bf02eaa: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::HeaderMap, std::__1::default_deleteEnvoy::Http::HeaderMap >&&, bool) + 11130
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: update landing #16 0x10bf062fa: non-virtual thunk to Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders(std::__1::unique_ptr<Envoy::Http::HeaderMap, std::__1::default_deleteEnvoy::Http::HeaderMap >&&, bool) + 58
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: route matching details #17 0x10bfdd282: Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete() + 418
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: cluster stats #18 0x10bfdd418: non-virtual thunk to Envoy::Http::Http1::ServerConnectionImpl::onMessageComplete() + 24
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: mongo proxy #19 0x10bfe1270: Envoy::Http::Http1::ConnectionImpl::$_7::operator()(http_parser*) const + 32
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Update what_is_envoy.rst #20 0x10bfe1247: Envoy::Http::Http1::ConnectionImpl::$_7::__invoke(http_parser*) + 23
[2018-05-09 11:50:07.236][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Fix typo #21 0x10bfeea43: http_parser_execute + 19235
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: SSL -> TLS #22 0x10bfda775: Envoy::Http::Http1::ConnectionImpl::dispatchSlice(char const*, unsigned long) + 101
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: README and getting help #23 0x10bfda635: Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) + 501
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Health Check Link #24 0x10bfdaad6: virtual thunk to Envoy::Http::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) + 38
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Change some sentences for clarity #25 0x10befa0fb: Envoy::Http::ConnectionManagerImpl::onData(Envoy::Buffer::Instance&, bool) + 1355
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Add docker example #26 0x10b9c6638: Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*) + 1496
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Return x-request-id when tracing enforced. #27 0x10b9c6801: Envoy::Network::FilterManagerImpl::onRead() + 177
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Use runtime key in duration filter. #28 0x10b9b516f: Envoy::Network::ConnectionImpl::onRead(unsigned long long) + 239
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Add tracing to obs on landing page. #29 0x10b9b81a5: Envoy::Network::ConnectionImpl::onReadReady() + 405
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: do ci on docs #30 0x10b9b7fc5: Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) + 789
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Change organization of the HTTP section of Architecture overview #31 0x10b9bef5d: Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, bool)::$_2::operator()(unsigned int) const + 29
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> request shadowing #32 0x10b9bef2e: _ZNSt3__128__invoke_void_return_wrapperIvE6__callIJRZN5Envoy7Network14ConnectionImplC1ERNS3_5Event10DispatcherEONS_10unique_ptrINS4_16ConnectionSocketENS_14default_deleteISA_EEEEONS9_INS4_15TransportSocketENSB_ISF_EEEEbE3$2jEEEvDpOT + 78
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> optimize getRawSlices() #33 0x10b9bee28: std::__1::__function::__func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, bool)::$_2, std::__1::allocator<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__1::unique_ptr<Envoy::Network::ConnectionSocket, std::__1::default_deleteEnvoy::Network::ConnectionSocket >&&, std::__1::unique_ptr<Envoy::Network::TransportSocket, std::__1::default_deleteEnvoy::Network::TransportSocket >&&, bool)::$_2>, void (unsigned int)>::operator()(unsigned int&&) + 56
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Request shadowing already present. #34 0x10b99d457: std::__1::function<void (unsigned int)>::operator()(unsigned int) const + 151
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Example configurations #35 0x10b99d3ac: Envoy::Event::FileEventImpl::assignEvents(unsigned int)::$_0::operator()(int, short, void*) const + 236
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Per zone rq (total, timing, per response code, etc) stats for upstream cluster. #36 0x10b99d2b7: Envoy::Event::FileEventImpl::assignEvents(unsigned int)::$_0::__invoke(int, short, void*) + 55
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> allow fast fail for travis build. #37 0x10c39d1da: event_process_active_single_queue + 1450
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: example configurations #38 0x10c399dad: event_base_loop + 1741
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> Differentiate in x-request-id value what made it to be traceable. #39 0x10b9956a1: Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) + 97
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> fixes #40 0x10b8d64ed: Envoy::Server::InstanceImpl::run() + 813
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: fixes #41 0x10b1e8326: Envoy::MainCommonBase::run() + 150
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:105] thr<0> docs: sd fix #42 0x10b1b7a1b: Envoy::MainCommon::run() + 27
[2018-05-09 11:50:07.237][1843493][critical][backtrace] bazel-out/darwin-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:110] end backtrace thread 0
Activity