Return x-request-id when tracing enforced. #27

RomanDzhabarov · 2016-08-21T05:56:04Z

No description provided.

mattklein123 · 2016-08-21T06:45:10Z

source/common/http/conn_manager_impl.cc

@@ -510,6 +509,11 @@ void ConnectionManagerImpl::ActiveStream::encodeHeaders(ActiveStreamEncoderFilte
                         connection_manager_.codec_->protocolString());
  ConnectionManagerUtility::mutateResponseHeaders(headers, connection_manager_.config_);

+  if (Tracing::HttpTracerUtility::isForcedTracing(*request_headers_)) {


Do this inside ConnectionManagerUtility::mutateResponseHeaders, much easier to test.

agree, changed

mattklein123 · 2016-08-21T22:23:06Z

source/common/tracing/http_tracer_impl.cc

@@ -55,6 +55,11 @@ Decision HttpTracerUtility::isTracing(const Http::AccessLog::RequestInfo& reques
  return {Reason::NotTraceableRequestId, false};
 }

+bool HttpTracerUtility::isForcedTracing(const Http::HeaderMap& request_headers) {
+  return request_headers.has(Http::Headers::get().EnvoyInternalRequest) &&
+         request_headers.has(Http::Headers::get().ForceTrace);


It's actually a bug that ForceTrace is not cleaned from external headers. You should clean it (and test for it) and the mutate request headers section in ConnManagerUtility. Then this check only needs to look for ForceTrace. IMO ForceTrace should also be cleaned entirely once it's processed (whether internal or external), to avoid propagating to the next hop, but that's a larger change.

fixed it (cleaning it on non internal requests).

although propagating is possible this will require service to read/copy specific header and pass it to another egress call.

Sorry I meant that right now if someone sets force-trace, we still send that header on, which will trigger force trace on the next hop. That hop may or may not have tracing enabled, and it doesn't really matter since we have already modified x-request-id, but technically I think if someone sets force trace header we should clean it no matter what after it is used. This is not a big deal.

mattklein123 · 2016-08-22T02:45:25Z

source/common/http/conn_manager_utility.cc

@@ -65,6 +66,7 @@ void ConnectionManagerUtility::mutateRequestHeaders(Http::HeaderMap& request_hea
    request_headers.remove(Headers::get().EnvoyUpstreamRequestTimeoutMs);
    request_headers.remove(Headers::get().EnvoyUpstreamRequestPerTryTimeoutMs);
    request_headers.remove(Headers::get().EnvoyExpectedRequestTimeoutMs);
+    request_headers.remove(Headers::get().ForceTrace);


nit: can you rename this "EnvoyForceTrace" (and reorder in headers.h). Not related to this review but might as well fix now. Otherwise looks good.

mattklein123 · 2016-08-22T02:45:40Z

👍 after variable name change

Improve envoy prototype

…ardown. server_fuzz_test indicated the below crash, where the DispatcherImpl teardown releases some upstream client SSL related objects that then needs SecretManagerImpl to unregister. Previously, this was already destructed by time we were in ~DispatcherImpl(), this PR reorders. #0 0xc1e826 in size /usr/local/include/c++/v1/__hash_table:809:55 envoyproxy#1 0xc1e826 in bucket_count /usr/local/include/c++/v1/__hash_table:1197 envoyproxy#2 0xc1e826 in std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, void*>*> std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> > > >::find<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/include/c++/v1/__hash_table:2334 envoyproxy#3 0xc1e278 in unsigned long std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> > > >::__erase_unique<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/include/c++/v1/__hash_table:2510:20 envoyproxy#4 0xc1def6 in erase /usr/local/include/c++/v1/unordered_map:1156:59 envoyproxy#5 0xc1def6 in Envoy::Secret::SecretManagerImpl::DynamicSecretProviders<Envoy::Secret::TlsCertificateSdsApi>::removeDynamicSecretProvider(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /bazel-out/k8-fastbuild/bin/source/common/secret/_virtual_includes/secret_manager_impl_lib/common/secret/secret_manager_impl.h:75 envoyproxy#6 0x724aa9 in Envoy::Cleanup::~Cleanup() /bazel-out/k8-fastbuild/bin/source/common/common/_virtual_includes/cleanup_lib/common/common/cleanup.h:11:16 envoyproxy#7 0xc1ff33 in Envoy::Secret::SdsApi::~SdsApi() /bazel-out/k8-fastbuild/bin/source/common/secret/_virtual_includes/sds_api_lib/common/secret/sds_api.h:29:7 envoyproxy#8 0xc188a0 in __release_shared /usr/local/include/c++/v1/memory:3530:9 envoyproxy#9 0xc188a0 in __release_shared /usr/local/include/c++/v1/memory:3572 envoyproxy#10 0xc188a0 in std::__1::shared_ptr<Envoy::Secret::SecretProvider<Envoy::Ssl::TlsCertificateConfig> >::~shared_ptr() /usr/local/include/c++/v1/memory:4508 envoyproxy#11 0x149c922 in Envoy::Ssl::ContextConfigImpl::~ContextConfigImpl() /source/common/ssl/context_config_impl.cc:117:1 envoyproxy#12 0x14a0d8f in Envoy::Ssl::ClientContextConfigImpl::~ClientContextConfigImpl() /bazel-out/k8-fastbuild/bin/source/common/ssl/_virtual_includes/context_config_lib/common/ssl/context_config_impl.h:91:7 envoyproxy#13 0x14a0dc8 in Envoy::Ssl::ClientContextConfigImpl::~ClientContextConfigImpl() /bazel-out/k8-fastbuild/bin/source/common/ssl/_virtual_includes/context_config_lib/common/ssl/context_config_impl.h:91:7 envoyproxy#14 0x149815b in operator() /usr/local/include/c++/v1/memory:2325:5 envoyproxy#15 0x149815b in reset /usr/local/include/c++/v1/memory:2638 envoyproxy#16 0x149815b in ~unique_ptr /usr/local/include/c++/v1/memory:2592 envoyproxy#17 0x149815b in Envoy::Ssl::ClientSslSocketFactory::~ClientSslSocketFactory() /bazel-out/k8-fastbuild/bin/source/common/ssl/_virtual_includes/ssl_socket_lib/common/ssl/ssl_socket.h:83 envoyproxy#18 0x14981c8 in Envoy::Ssl::ClientSslSocketFactory::~ClientSslSocketFactory() /bazel-out/k8-fastbuild/bin/source/common/ssl/_virtual_includes/ssl_socket_lib/common/ssl/ssl_socket.h:83:7 envoyproxy#19 0x1362caf in operator() /usr/local/include/c++/v1/memory:2325:5 envoyproxy#20 0x1362caf in reset /usr/local/include/c++/v1/memory:2638 envoyproxy#21 0x1362caf in ~unique_ptr /usr/local/include/c++/v1/memory:2592 envoyproxy#22 0x1362caf in Envoy::Upstream::ClusterInfoImpl::~ClusterInfoImpl() /bazel-out/k8-fastbuild/bin/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:362 envoyproxy#23 0x1362d28 in Envoy::Upstream::ClusterInfoImpl::~ClusterInfoImpl() /bazel-out/k8-fastbuild/bin/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:362:7 envoyproxy#24 0x66e560 in __release_shared /usr/local/include/c++/v1/memory:3530:9 envoyproxy#25 0x66e560 in __release_shared /usr/local/include/c++/v1/memory:3572 envoyproxy#26 0x66e560 in std::__1::shared_ptr<Envoy::Upstream::ClusterInfo const>::~shared_ptr() /usr/local/include/c++/v1/memory:4508 envoyproxy#27 0x13621bf in Envoy::Upstream::HostImpl::~HostImpl() /bazel-out/k8-fastbuild/bin/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:156:7 envoyproxy#28 0x13621f8 in Envoy::Upstream::HostImpl::~HostImpl() /bazel-out/k8-fastbuild/bin/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:156:7 envoyproxy#29 0x66e650 in __release_shared /usr/local/include/c++/v1/memory:3530:9 envoyproxy#30 0x66e650 in __release_shared /usr/local/include/c++/v1/memory:3572 envoyproxy#31 0x66e650 in std::__1::shared_ptr<Envoy::Upstream::HostDescription const>::~shared_ptr() /usr/local/include/c++/v1/memory:4508 envoyproxy#32 0x13b20c3 in Envoy::Http::CodecClient::~CodecClient() /source/common/http/codec_client.cc:38:30 envoyproxy#33 0x13b2258 in Envoy::Http::CodecClientProd::~CodecClientProd() /bazel-out/k8-fastbuild/bin/source/common/http/_virtual_includes/codec_client_lib/common/http/codec_client.h:229:7 envoyproxy#34 0x751de6 in operator() /usr/local/include/c++/v1/memory:2325:5 envoyproxy#35 0x751de6 in reset /usr/local/include/c++/v1/memory:2638 envoyproxy#36 0x751de6 in ~unique_ptr /usr/local/include/c++/v1/memory:2592 envoyproxy#37 0x751de6 in destroy /usr/local/include/c++/v1/memory:1867 envoyproxy#38 0x751de6 in __destroy<std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> > > /usr/local/include/c++/v1/memory:1729 envoyproxy#39 0x751de6 in destroy<std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> > > /usr/local/include/c++/v1/memory:1597 envoyproxy#40 0x751de6 in __destruct_at_end /usr/local/include/c++/v1/vector:422 envoyproxy#41 0x751de6 in clear /usr/local/include/c++/v1/vector:365 envoyproxy#42 0x751de6 in std::__1::__vector_base<std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> >, std::__1::allocator<std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> > > >::~__vector_base() /usr/local/include/c++/v1/vector:459 envoyproxy#43 0x74d1aa in ~vector /usr/local/include/c++/v1/vector:551:5 envoyproxy#44 0x74d1aa in Envoy::Event::DispatcherImpl::~DispatcherImpl() /source/common/event/dispatcher_impl.cc:41 envoyproxy#45 0x74d658 in Envoy::Event::DispatcherImpl::~DispatcherImpl() /source/common/event/dispatcher_impl.cc:41:35 envoyproxy#46 0x697b76 in operator() /usr/local/include/c++/v1/memory:2325:5 envoyproxy#47 0x697b76 in reset /usr/local/include/c++/v1/memory:2638 envoyproxy#48 0x697b76 in ~unique_ptr /usr/local/include/c++/v1/memory:2592 envoyproxy#49 0x697b76 in Envoy::Server::InstanceImpl::InstanceImpl(Envoy::Server::Options&, Envoy::Event::TimeSystem&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::TestHooks&, Envoy::Server::HotRestart&, Envoy::Stats::StoreRoot&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGenerator, std::__1::default_delete<Envoy::Runtime::RandomGenerator> >&&, Envoy::ThreadLocal::Instance&) /source/server/server.cc:92 envoyproxy#50 0x586026 in make_unique<Envoy::Server::InstanceImpl, testing::NiceMock<Envoy::Server::MockOptions> &, Envoy::Event::TestTimeSystem &, std::__1::shared_ptr<Envoy::Network::Address::Ipv4Instance>, Envoy::DefaultTestHooks &, testing::NiceMock<Envoy::Server::MockHotRestart> &, Envoy::Stats::TestIsolatedStoreImpl &, Envoy::Thread::MutexBasicLockable &, Envoy::Server::TestComponentFactory &, std::__1::unique_ptr<Envoy::Runtime::RandomGeneratorImpl, std::__1::default_delete<Envoy::Runtime::RandomGeneratorImpl> >, Envoy::ThreadLocal::InstanceImpl &> /usr/local/include/c++/v1/memory:3118:32 envoyproxy#51 0x586026 in Envoy::Server::TestOneProtoInput(envoy::config::bootstrap::v2::Bootstrap const&) /test/server/server_fuzz_test.cc:78 Risk Level: Low Testing: Corpus entry added. It's pretty hard to build regressions for this kind of destruction ordering, so relying on server_fuzz_test + corpus. Signed-off-by: Harvey Tuch <[email protected]>

…ardown. (#4940) server_fuzz_test indicated the below crash, where the DispatcherImpl teardown releases some upstream client SSL related objects that then needs SecretManagerImpl to unregister. Previously, this was already destructed by time we were in ~DispatcherImpl(), this PR reorders. #0 0xc1e826 in size /usr/local/include/c++/v1/__hash_table:809:55 #1 0xc1e826 in bucket_count /usr/local/include/c++/v1/__hash_table:1197 #2 0xc1e826 in std::__1::__hash_iterator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, void*>*> std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> > > >::find<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/include/c++/v1/__hash_table:2334 #3 0xc1e278 in unsigned long std::__1::__hash_table<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::__unordered_map_hasher<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::__unordered_map_equal<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::weak_ptr<Envoy::Secret::TlsCertificateSdsApi> > > >::__erase_unique<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/include/c++/v1/__hash_table:2510:20 #4 0xc1def6 in erase /usr/local/include/c++/v1/unordered_map:1156:59 #5 0xc1def6 in Envoy::Secret::SecretManagerImpl::DynamicSecretProviders<Envoy::Secret::TlsCertificateSdsApi>::removeDynamicSecretProvider(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /bazel-out/k8-fastbuild/bin/source/common/secret/_virtual_includes/secret_manager_impl_lib/common/secret/secret_manager_impl.h:75 #6 0x724aa9 in Envoy::Cleanup::~Cleanup() /bazel-out/k8-fastbuild/bin/source/common/common/_virtual_includes/cleanup_lib/common/common/cleanup.h:11:16 #7 0xc1ff33 in Envoy::Secret::SdsApi::~SdsApi() /bazel-out/k8-fastbuild/bin/source/common/secret/_virtual_includes/sds_api_lib/common/secret/sds_api.h:29:7 #8 0xc188a0 in __release_shared /usr/local/include/c++/v1/memory:3530:9 #9 0xc188a0 in __release_shared /usr/local/include/c++/v1/memory:3572 #10 0xc188a0 in std::__1::shared_ptr<Envoy::Secret::SecretProvider<Envoy::Ssl::TlsCertificateConfig> >::~shared_ptr() /usr/local/include/c++/v1/memory:4508 #11 0x149c922 in Envoy::Ssl::ContextConfigImpl::~ContextConfigImpl() /source/common/ssl/context_config_impl.cc:117:1 #12 0x14a0d8f in Envoy::Ssl::ClientContextConfigImpl::~ClientContextConfigImpl() /bazel-out/k8-fastbuild/bin/source/common/ssl/_virtual_includes/context_config_lib/common/ssl/context_config_impl.h:91:7 #13 0x14a0dc8 in Envoy::Ssl::ClientContextConfigImpl::~ClientContextConfigImpl() /bazel-out/k8-fastbuild/bin/source/common/ssl/_virtual_includes/context_config_lib/common/ssl/context_config_impl.h:91:7 #14 0x149815b in operator() /usr/local/include/c++/v1/memory:2325:5 #15 0x149815b in reset /usr/local/include/c++/v1/memory:2638 #16 0x149815b in ~unique_ptr /usr/local/include/c++/v1/memory:2592 #17 0x149815b in Envoy::Ssl::ClientSslSocketFactory::~ClientSslSocketFactory() /bazel-out/k8-fastbuild/bin/source/common/ssl/_virtual_includes/ssl_socket_lib/common/ssl/ssl_socket.h:83 #18 0x14981c8 in Envoy::Ssl::ClientSslSocketFactory::~ClientSslSocketFactory() /bazel-out/k8-fastbuild/bin/source/common/ssl/_virtual_includes/ssl_socket_lib/common/ssl/ssl_socket.h:83:7 #19 0x1362caf in operator() /usr/local/include/c++/v1/memory:2325:5 #20 0x1362caf in reset /usr/local/include/c++/v1/memory:2638 #21 0x1362caf in ~unique_ptr /usr/local/include/c++/v1/memory:2592 #22 0x1362caf in Envoy::Upstream::ClusterInfoImpl::~ClusterInfoImpl() /bazel-out/k8-fastbuild/bin/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:362 #23 0x1362d28 in Envoy::Upstream::ClusterInfoImpl::~ClusterInfoImpl() /bazel-out/k8-fastbuild/bin/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:362:7 #24 0x66e560 in __release_shared /usr/local/include/c++/v1/memory:3530:9 #25 0x66e560 in __release_shared /usr/local/include/c++/v1/memory:3572 #26 0x66e560 in std::__1::shared_ptr<Envoy::Upstream::ClusterInfo const>::~shared_ptr() /usr/local/include/c++/v1/memory:4508 #27 0x13621bf in Envoy::Upstream::HostImpl::~HostImpl() /bazel-out/k8-fastbuild/bin/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:156:7 #28 0x13621f8 in Envoy::Upstream::HostImpl::~HostImpl() /bazel-out/k8-fastbuild/bin/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:156:7 #29 0x66e650 in __release_shared /usr/local/include/c++/v1/memory:3530:9 #30 0x66e650 in __release_shared /usr/local/include/c++/v1/memory:3572 #31 0x66e650 in std::__1::shared_ptr<Envoy::Upstream::HostDescription const>::~shared_ptr() /usr/local/include/c++/v1/memory:4508 #32 0x13b20c3 in Envoy::Http::CodecClient::~CodecClient() /source/common/http/codec_client.cc:38:30 #33 0x13b2258 in Envoy::Http::CodecClientProd::~CodecClientProd() /bazel-out/k8-fastbuild/bin/source/common/http/_virtual_includes/codec_client_lib/common/http/codec_client.h:229:7 #34 0x751de6 in operator() /usr/local/include/c++/v1/memory:2325:5 #35 0x751de6 in reset /usr/local/include/c++/v1/memory:2638 #36 0x751de6 in ~unique_ptr /usr/local/include/c++/v1/memory:2592 #37 0x751de6 in destroy /usr/local/include/c++/v1/memory:1867 #38 0x751de6 in __destroy<std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> > > /usr/local/include/c++/v1/memory:1729 #39 0x751de6 in destroy<std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> > > /usr/local/include/c++/v1/memory:1597 #40 0x751de6 in __destruct_at_end /usr/local/include/c++/v1/vector:422 #41 0x751de6 in clear /usr/local/include/c++/v1/vector:365 #42 0x751de6 in std::__1::__vector_base<std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> >, std::__1::allocator<std::__1::unique_ptr<Envoy::Event::DeferredDeletable, std::__1::default_delete<Envoy::Event::DeferredDeletable> > > >::~__vector_base() /usr/local/include/c++/v1/vector:459 #43 0x74d1aa in ~vector /usr/local/include/c++/v1/vector:551:5 #44 0x74d1aa in Envoy::Event::DispatcherImpl::~DispatcherImpl() /source/common/event/dispatcher_impl.cc:41 #45 0x74d658 in Envoy::Event::DispatcherImpl::~DispatcherImpl() /source/common/event/dispatcher_impl.cc:41:35 #46 0x697b76 in operator() /usr/local/include/c++/v1/memory:2325:5 #47 0x697b76 in reset /usr/local/include/c++/v1/memory:2638 #48 0x697b76 in ~unique_ptr /usr/local/include/c++/v1/memory:2592 #49 0x697b76 in Envoy::Server::InstanceImpl::InstanceImpl(Envoy::Server::Options&, Envoy::Event::TimeSystem&, std::__1::shared_ptr<Envoy::Network::Address::Instance const>, Envoy::TestHooks&, Envoy::Server::HotRestart&, Envoy::Stats::StoreRoot&, Envoy::Thread::BasicLockable&, Envoy::Server::ComponentFactory&, std::__1::unique_ptr<Envoy::Runtime::RandomGenerator, std::__1::default_delete<Envoy::Runtime::RandomGenerator> >&&, Envoy::ThreadLocal::Instance&) /source/server/server.cc:92 #50 0x586026 in make_unique<Envoy::Server::InstanceImpl, testing::NiceMock<Envoy::Server::MockOptions> &, Envoy::Event::TestTimeSystem &, std::__1::shared_ptr<Envoy::Network::Address::Ipv4Instance>, Envoy::DefaultTestHooks &, testing::NiceMock<Envoy::Server::MockHotRestart> &, Envoy::Stats::TestIsolatedStoreImpl &, Envoy::Thread::MutexBasicLockable &, Envoy::Server::TestComponentFactory &, std::__1::unique_ptr<Envoy::Runtime::RandomGeneratorImpl, std::__1::default_delete<Envoy::Runtime::RandomGeneratorImpl> >, Envoy::ThreadLocal::InstanceImpl &> /usr/local/include/c++/v1/memory:3118:32 #51 0x586026 in Envoy::Server::TestOneProtoInput(envoy::config::bootstrap::v2::Bootstrap const&) /test/server/server_fuzz_test.cc:78 Fixes oss-fuzz issue https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=11231 Risk Level: Low Testing: Corpus entry added. It's pretty hard to build regressions for this kind of destruction ordering, so relying on server_fuzz_test + corpus. Signed-off-by: Harvey Tuch <[email protected]>

sync to latest

…_cache.cc This avoids a deadlock when a library which is being dlopen'ed creates as part of its static constructors a thread which quickly need to call malloc. We are still in the dlopen call (so with some internal glibc mutex taken) when the thread executes code and later needs to call malloc which in term calls tls_get_addr_tail, which wait for the dlopen mutex to be unlocked. If later the dlopen'ing thread also calls malloc as part of its constructors, we are in a deadlock. Fix is similar to gperftools/gperftools@7852eeb Stack of the dlopening thread: #0 0x00007fd5406ca93c in __lll_lock_wait () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0 #1 0x00007fd5406c45a5 in pthread_mutex_lock () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0 ... proprietary code in the stack envoyproxy#9 0x00007fd5074f0367 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at src/ClientImpl.cpp:15 envoyproxy#10 0x00007fd5074f06d7 in _GLOBAL__sub_I_ClientImpl.cpp(void) () at src/ClientImpl.cpp:85 envoyproxy#11 0x00007fd50757aa46 in __do_global_ctors_aux () envoyproxy#12 0x00007fd5073e985f in _init () from ... envoyproxy#13 0x00007fd53bf9dec8 in ?? () from ... envoyproxy#14 0x00007fd54d637a5d in call_init.part () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2 envoyproxy#15 0x00007fd54d637bab in _dl_init () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2 envoyproxy#16 0x00007fd54d63c160 in dl_open_worker () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2 envoyproxy#17 0x00007fd54d637944 in _dl_catch_error () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2 envoyproxy#18 0x00007fd54d63b7d9 in _dl_open () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2 envoyproxy#19 0x00007fd54d61f2b9 in dlopen_doit () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2 envoyproxy#20 0x00007fd54d637944 in _dl_catch_error () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2 envoyproxy#21 0x00007fd54d61f889 in _dlerror_run () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2 envoyproxy#22 0x00007fd54d61f351 in dlopen@@GLIBC_2.2.5 () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libdl.so.2 Stack of the newly created thread calling tls_get_addr_tail: #0 0x00007fd5406ca93c in __lll_lock_wait () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0 #1 0x00007fd5406c4622 in pthread_mutex_lock () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0 #2 0x00007fd54d63a2ed in tls_get_addr_tail () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/ld-linux-x86-64.so.2 envoyproxy#3 0x00007fd53fee877d in tcmalloc::ThreadCache::CreateCacheIfNecessary () at src/thread_cache.cc:344 envoyproxy#4 0x00007fd53fecb4ab in tcmalloc::ThreadCache::GetCache () at src/thread_cache.h:437 envoyproxy#5 0x00007fd53fefeccb in (anonymous namespace)::do_malloc (size=56) at src/tcmalloc.cc:1354 envoyproxy#6 tcmalloc::do_allocate_full<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1762 envoyproxy#7 tcmalloc::allocate_full_cpp_throw_oom (size=56) at src/tcmalloc.cc:1776 envoyproxy#8 0x00007fd53ff01b80 in tcmalloc::dispatch_allocate_full<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1785 envoyproxy#9 malloc_fast_path<tcmalloc::cpp_throw_oom> (size=56) at src/tcmalloc.cc:1845 envoyproxy#10 tc_new (size=56) at src/tcmalloc.cc:1980 ... proprietary code in the stack envoyproxy#26 0x00007fd5406c1ef4 in start_thread () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libpthread.so.0 envoyproxy#27 0x00007fd5403ba01d in clone () from /data3/mwrep/rgeissler/core.tls/opt/1A/toolchain/x86_64-2.6.32-v2/lib64/libc.so.6

RomanDzhabarov added 2 commits August 20, 2016 22:31

x-request-id preparation work

1027357

Add tests for isForcedTracing.

574acca

RomanDzhabarov changed the title ~~Return x-request-id when forcing enforced.~~ Return x-request-id when tracing enforced. Aug 21, 2016

mattklein123 reviewed Aug 21, 2016
View reviewed changes

RomanDzhabarov and others added 3 commits August 21, 2016 13:45

Move logic into mutateResponseHeaders.

e899995

revert source/common/http/conn_manager_impl.h.

31a4b25

Update headers.rst

98d4672

mattklein123 reviewed Aug 21, 2016
View reviewed changes

RomanDzhabarov added 3 commits August 21, 2016 16:06

Clean x-envoy-force-trace header on not internal requests.

ccfe069

Merge branch 'return-rq' of github.com:lyft/envoy into return-rq

7c5ef2b

revert source/common/tracing/http_tracer_impl.h

ac25700

mattklein123 reviewed Aug 22, 2016
View reviewed changes

s/ForceTrace/EnvoyForceTrace.

e9ca23a

RomanDzhabarov merged commit d1c5c10 into master Aug 22, 2016

RomanDzhabarov deleted the return-rq branch August 22, 2016 03:27

andrewvd mentioned this pull request Jun 19, 2017

SIGSEGV Segmentation fault #1138

Closed

phanama mentioned this pull request Feb 28, 2018

Envoy Crashes at 300+ req/s using hot-restarter and systemd #2685

Closed

shuoli84 mentioned this pull request May 9, 2018

crashed when click config_dump #3329

Closed

lap1817 mentioned this pull request Jun 15, 2018

TCP_Proxy: onLowWatermark crash #3639

Closed

qiwzhang mentioned this pull request Jul 25, 2018

Crash at RDS onConfigUpdate #3953

Closed

qiwzhang mentioned this pull request Aug 15, 2018

GrpcSubscriptionImpl bug: send another request at destructor #4167

Closed

rshriram pushed a commit to rshriram/envoy that referenced this pull request Oct 30, 2018

Merge pull request envoyproxy#27 from lizan/fix_prototype_memory

e941522

Improve envoy prototype

alyssawilk mentioned this pull request Nov 26, 2018

"pure virtual method called" in grpc:grpc_client_integration_test #5071

Closed

alyssawilk mentioned this pull request Dec 13, 2018

throw_bad_optional_access in HttpConnectionManager #5289

Closed

zyfjeff mentioned this pull request Dec 15, 2018

Crash due to incorrect object destruction order #5313

Closed

wgallagher mentioned this pull request Apr 7, 2020

core dump in AccessLog::RuntimeFilter when RequestIDExtension not set #10686

Closed

HenryYYang mentioned this pull request Jun 5, 2020

Fix race condition when the redis filter is destroyed. #11466

Merged

alyssawilk mentioned this pull request Jul 1, 2020

LoadStatsIntegrationTest flake #11784

Closed

antoniovicente mentioned this pull request Jul 23, 2020

dispatcher: Run zero-delay timeout timers on the next iteration of the event loop #11823

Merged

antoniovicente mentioned this pull request Aug 3, 2020

tcp tunneling integration flake #12253

Closed

raonitimo mentioned this pull request Aug 5, 2020

Envoy crashes with "assert failure: !api_state.paused_." #12488

Closed

incfly mentioned this pull request Sep 23, 2020

Data race in callback manager / ClusterInfoImpl destruction #13209

Closed

antoniovicente mentioned this pull request Nov 19, 2020

Test flake in OauthIntegrationTest.UnauthenticatedFlow #12960

Closed

danzh2010 mentioned this pull request Mar 10, 2021

quiche: TSAN failure in quic_protocol_integration_test #15414

Closed

davinci26 pushed a commit to davinci26/envoy that referenced this pull request Mar 26, 2021

Merge pull request envoyproxy#27 from envoyproxy/master

61e51ed

sync to latest

antoniovicente mentioned this pull request May 14, 2021

QuicHttpIntegrationTest fails about 20% of the time under clang-tsan due to FakeUpstream object lifetime problems #16493

Closed

ayyatsenko mentioned this pull request Jun 25, 2021

Catch Segmentation Fault using UpstreamStartTls in network filter #17144

Closed

danzh2010 mentioned this pull request Nov 3, 2021

multiplexed_integration_test flaky #18889

Closed

yanjunxiang-google mentioned this pull request Apr 20, 2022

MemorySanitizer: use-of-uninitialized-value issue due to accessing destroyed class member #20913

Merged

RyanTheOptimist mentioned this pull request Jun 2, 2022

TSAN error in h1_capture_persistent_fuzz_test #21547

Closed

huanghuangzym mentioned this pull request Jun 30, 2022

Segfault from ECDS update #14930

Closed

yanjunxiang-google mentioned this pull request Nov 1, 2022

fixing OSS filter_fuzz_test OOM issue when thread_count is 800 million. #23782

Merged

sharmamona mentioned this pull request Nov 18, 2022

RELEASE_ASSERT when creating upstream connection and socket call failed with ENOBUFS #24073

Open

alyssawilk mentioned this pull request Mar 20, 2023

flake in IpVersions/GolangIntegrationTest.LocalReply_DecodeHeader/IPv6 #26182

Closed

yanjunxiang-google mentioned this pull request May 8, 2023

Envoy ext_proc filter throw exception when received response timeout Duration is too large #27260

Merged

yanjunxiang-google mentioned this pull request May 22, 2023

Assert(valid) when ext_proc filter apply header mutations #27547

Merged

yanjunxiang-google mentioned this pull request May 30, 2023

ext_proc fuzzer test trigger ENVOY_BUG when clear route cache #27657

Merged

yanjunxiang-google mentioned this pull request Jul 25, 2023

ext_proc filter ASSERT at canIterate() when client not sending trailer #28592

Merged

yanjunxiang-google mentioned this pull request Nov 27, 2023

Revert ext_proc: send attributes #30781 #31017

Merged

danzh2010 mentioned this pull request Mar 15, 2024

quic integration test Asan violation #32340

Open

juanmolle mentioned this pull request Mar 26, 2024

POC wasm as upstream filter #33119

Closed

arminabf pushed a commit to arminabf/envoy that referenced this pull request Jun 5, 2024

Add buildtool dockerimage of wasi-sdk (envoyproxy#27)

9237dd7

dceravigupta mentioned this pull request Jul 8, 2024

Envoy validate mode not working with envoy.access_loggers.fluentd extension #35098

Closed

Bikara950225 mentioned this pull request Nov 18, 2024

When I use the admin_config configuration in Tap, Envoy fails to start. #37195

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Return x-request-id when tracing enforced. #27

Return x-request-id when tracing enforced. #27

RomanDzhabarov commented Aug 21, 2016

mattklein123 Aug 21, 2016

RomanDzhabarov Aug 21, 2016

mattklein123 Aug 21, 2016

RomanDzhabarov Aug 22, 2016

mattklein123 Aug 22, 2016

mattklein123 Aug 22, 2016

mattklein123 commented Aug 22, 2016

Return x-request-id when tracing enforced. #27

Return x-request-id when tracing enforced. #27

Conversation

RomanDzhabarov commented Aug 21, 2016

mattklein123 Aug 21, 2016

Choose a reason for hiding this comment

RomanDzhabarov Aug 21, 2016

Choose a reason for hiding this comment

mattklein123 Aug 21, 2016

Choose a reason for hiding this comment

RomanDzhabarov Aug 22, 2016

Choose a reason for hiding this comment

mattklein123 Aug 22, 2016

Choose a reason for hiding this comment

mattklein123 Aug 22, 2016

Choose a reason for hiding this comment

mattklein123 commented Aug 22, 2016