[Notes] [Git][BuildStream/buildstream][danielsilverstone-ct/further-optimisations] 2 commits: _cachekey.py: Add a variant of the calculator which take pre-sanitized input

[Daniel Silverstone <gitlab mg gitlab com>]

Title: GitLab

Daniel Silverstone pushed to branch danielsilverstone-ct/further-optimisations at BuildStream / buildstream

Commits:

• 383ae15c
  by Daniel Silverstone at 2018-11-08T14:27:26Z

  _cachekey.py: Add a variant of the calculator which take pre-sanitized input
  
  In order to support a case where the caller already has pre-sanitized input
  this variant of the cache key generator needs to exist.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• d0485038
  by Daniel Silverstone at 2018-11-08T14:29:56Z

  element.py: Alter Element.__calculate_cache_key() to pre-santize inputs
  
  In order to reduce the effort we spend in `_yaml.node_sanitize()` take
  advantage of how `OrderedDict` works and pre-sanitize the majority of
  the cache-key once.  This approximately halves the amount of effort we
  spend in `_yaml.node_sanitize()` in pre-scheduler build scenarios.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

2 changed files:

• buildstream/_cachekey.py
• buildstream/element.py

Changes:

buildstream/_cachekey.py

---

...	...	@@ -40,3 +40,19 @@ def generate_key(value):
40	40	ordered = _yaml.node_sanitize(value)
41	41	string = pickle.dumps(ordered)
42	42	return hashlib.sha256(string).hexdigest()
	43	+
	44	+# generate_key_pre_sanitized()
	45	+#
	46	+# Generate an sha256 hex digest from the given value. The value
	47	+# must be (a) compatible with generate_key() and (b) already have
	48	+# been passed through _yaml.node_sanitize()
	49	+#
	50	+# Args:
	51	+# value: A sanitized value to get a key for
	52	+#
	53	+# Returns:
	54	+# (str): An sha256 hex digest of the given value
	55	+#
	56	+def generate_key_pre_sanitized(value):
	57	+ string = pickle.dumps(value)
	58	+ return hashlib.sha256(string).hexdigest()

buildstream/element.py

---

...	...	@@ -2060,11 +2060,14 @@ class Element(Plugin):
2060	2060	}
2061	2061
2062	2062	self.__cache_key_dict['fatal-warnings'] = sorted(project._fatal_warnings)
	2063	+ self.__cache_key_dict['dependencies'] = []
	2064	+ self.__cache_key_dict = _yaml.node_sanitize(self.__cache_key_dict)
2063	2065
2064		- cache_key_dict = self.__cache_key_dict.copy()
2065		- cache_key_dict['dependencies'] = dependencies
	2066	+ # This replacement is safe since OrderedDict replaces the value,
	2067	+ # leaving its location in the dictionary alone.
	2068	+ self.__cache_key_dict['dependencies'] = _yaml.node_sanitize(dependencies)
2066	2069
2067		- return _cachekey.generate_key(cache_key_dict)
	2070	+ return _cachekey.generate_key_pre_sanitized(self.__cache_key_dict)
2068	2071
2069	2072	# __can_build_incrementally()
2070	2073	#

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.